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CHAPTER I 



THE PROBLEM AND RESEARCH METHOD 



One of the most serious factors hampering the library profession 
in its attempt to deal with the present crisis in library personnel 
is a lack of objective knowledge- -and thus of substantial agreement-- 
about the qualifications and characteristics necessary for effective 
service in various library positions. On the one hand} library educators 
find it difficult to arrive at judgments as to what should be taught 
prospective librarians when so little is known about the true character- 
istics of the market for their product, especially in terms of the real 
differences between the tasks the professionally educated librarians 
will be asked to perform and those tasks assigned to staff lacking 
such education. On the other hand, library administrators, who are 
charged by the educators with mitigating this condition by careful 
study of job requirements and more enlightened assignments of talent, 
feel greatly hampered by a lack of objective evidence of a real re- 
lationship between library education and effective performance at 
specific job levels. This leaves the way open for some administrators 
to assert that not enough differences in capabilities between trained 
and untrained people exist to make the differential assignments to 
staff called for by the educators and to lay the fault for this 
situation on mis judgments by library educators.^ 

The reference staffing aspect of this more general problem is 
particularly challenging. For one thing, there is a fair-sized body 
of speculative and pragmatic literature, based on the experience of 
reference librarians and administrators, concerning the qualifications 
needed for referexice work. And yet, this literature raises serious 
problems. One group of writers, noting that it is in reference work 
that the library staff comes into most crucial direct contact with the 
patron, emphasizes native qualities and personality traits, with 
practically every possible desirable human trait having at one time or 
another been put forward as necessary to effective reference work. 

On the other hand, there is an equally strong theme in the literature 
to the effect that extensive scholarly and education attainments are of 
utmost importance to reference work.^ Since all these qualities are 
surely not to be found in any but a very few persons, the literature 
raises the problem of relative importance between native qualities, 
presently very hard to identify and measure, and educational attain- 
ments, as yet not specified in much useful detail. 
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Reference service is all the more interesting in that it provides 
an assumption or normative principle as a point at which objective 
research might begin* Since one of the main tasks of library reference 
work is that of answering reference questions asked by patrons, it 
might be expected that differences on the variable of the reference 
question, particularly the dimension of difficulty, would be used 
in making decisions concerning differential staff qualifications and 
assignments* However, as yet the practitioners and administrators 
have for many practical and theoretical reasons had very little 
success in producing even widely agreed upon and useful definitions 
for reference questions, much less in characterizing and measuring 
differences in difficulty among them*^ This, plus the fact that 
questions of varying difficulty come to the reference desk with no 
predictability, making staff assignment on the basis of question 
difficulty all the more hard, has over the years helped to produce 
the assumption or norm that all reference work involving answering 
patrons' questions (except perhaps the most obviously simple 
"directional” questions) requires the qualifications of a professionally 
trained librarian*^ 

The general problem set for this study was to gain some empirical 
data which might bear on the factual basis for this assumption or norm* 
More specifically, the study was aimed at exploring the relationship 
bet\?een formal library training and effectiveness in answering reference 
questions* 



Related Literature 

There is a considerable body of literature reporting studies on 
the relationship between formal education and occupational success 
in various fields, including a few in librarianship* McCrossan has 
surveyed the major studies among these *^ He found that the better 
studies showed small or moderate relationships between education and 
job success and concluded that further research should show a similar 
relationship between library school education and competence in 
specific areas of library practice*^ 

In the specific area of reference work the few studies touching on 
personnel are part of a relatively small body of research literature 
on reference service in general* The existence of several reviews and 
surveys of this literature makes repetition here unnecessary* Perhaps 
the best such review article is that by Roths r.ein,® who in turn cites 
Shores^ and the chapter by Wheeler and Goldhor^® as among the most 
comprehensive and discriminating sunmaries* 

While most of the studies dealing with educational requirements 
for reference personnel have started with the very assumption that is 
under investigation in the present study, a few have attempted to 
provide information dealing with the value or necessity of formal 
library education for reference work* Breed attempted to ascertain the 
extent to which certain kinds of knowledge were called upon by 
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reference librarians in the search process used for answering ques- 
tions.^^ He found that 81 percent of the decisions or process steps 
taken by the librarians studied relied on library specialization 
knowledge. However, since he included in the library specialization 
category knowledge derived from library work experience as well as 
that gained from formal training, little information is provided by 
the study on the question of the value of formal education. 

In a study at the John Crerar Library, Jacobs concluded that 
around one-half of the questions received by the Science and Tech- 
nology Department could have been answered by personnel lacking 
formal library training. However, rather than actually measuring 
the ability of untrained persons to answer the questions received by 
the department, Jacobs used a priori definitions of the kinds of 
questions among all those received that would be appropriate to 
untrained staff. Most of the kinds of questions included among. those 
considered answerable by untrained personnel pertained to lending 
materials, orders for photoduplications, and simple directional 
questions. Since these question types are usually excluded from 
those called reference questions in most libraries, little data on 
the central problem of the present study was provided. 

Finally, Guerrier studied the reference questions asked in nine 
large city public library systems during one week in 1935. ^ Although 
the main purpose of her study was to provide information on the 
measurement and tabulation of reference work, her conclusions did 
touch on reference staffing. After designing a four-category clas- 
sification and grouping the questions studied, Guerrier concluded: 

The analysis of questions submitted by libraries in 
different sections of the United States emphasizes 
the fact that the major part of what is termed 
"reference work" is of the fact-finding or infor- 
mation type. In answering such questions the 
services of a specialist are not required. It 
should be possible for a general assistant with library 
school or college education who thinks and acts 
quickly and who is endowed with common sense, a 
retentive memory and fact-collecting instincts to 
become in a short time qualified to answer without 
fumbling 83% of the questions submitted. . . .^^ 

(Italics added) 

Guerrier *s study provided data only on the frequency of the types of 
questions, not on the ability of general assistants with college 
education to answer them; the latter seemingly came from the inves- 
tigator* s extensive library experience. 



The Hypothesis 

In an attempt to specify more explicitly the relationship between 
formal library education and reference performance, as a step in 
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formalizing the assumption or norm mentioned above into a ® 

to guide the present study, other parts of the literature of referenc 
work, particularly those dealing with reference 
reference administration, were examined. By noting what libr y 
school teachers generally consider important for a student to lea , 
relative to reference work, and what reference administrators expect 
the reference librarian to bring from library school, ^the assumed 
relationship between education and performance might be more pre- 
cisely defined. 

Central to reference instruction has always been an emphasis on 
the student's gaining control over as much information as possible, 
in order that he might bring this information to bear on the needs of 
his patrons. In earlier times it was thought that this could best 
be achieved merely by using training and experience to give the 
reference librarian direct and detailed acquaintance with as many 
information sources, or "reference" materials as possible. In the 
last few decades the feeling has developed that gaining control oyer 
or access to the information needed to answer the reference questions 
of library patrons requires a somewhat different kind of knowledge 

and training. 

In this newer concept of reference education what is consist- 
ently emphasized is that the aspiring reference librarian must bring 
to library school a broad general education; he must learn the 
existence and characteristics of a number of reference tools, par- 
ticularly those serving as keys to the information content of the 
rest of the collection; and he must use these two, in combination 
with concepts and information from other library school courses, 
such as cataloging, to acquire skill in certain techniques for 
answering reference questions. This reference skill, which is to 
be the product of the reference librarian s education, has been 
characterized as a reasoning process, 16 as a process of classi- 
fication or categorization, 17 or as a decision-making process 
Lscribable in tL language of decision theory or information science. 8 
Generally speaking, this process consists of: 1) making sure the 

question, as understood by the librarian, coincides with the actual 
Lformation needs of the patron; 2) analyzing, clas- 

sifying the question on a number of dimensions in order to formulate 
a search strategy; 3) translating the terms of the question into the 
language of the relevant parts of the reference system (catalog 
subject headings, bibliography entries, index terms, etc.); 4) making 
various decisions involved in conducting the search itself; and 
5) evaluating the information found, in terms of the patron and his 

needs. 



The authors of the literature on reference work do not imply any 
direct one-to-one relationship between library school training and 
success as a reference librarian, particularly over time after the 
librarian's graduation. Rather, the implication seems to be that the 
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background brought to library school; the tools, skills, and general 
approaches learned there; and the subsequent structuring of reference 
experience will combine to produce the skillful use of a general 
reference approach that enables the trained librarian effectively to 
bring under his control the relevant portions of a large body of 
Information to answer the question at hand. 

Though It Is never explicitly stated, the correlative Implication 
seems to be that the skill of reference librarians without profes- 
sional training will be based on learning by experience that certain 
documents from a given collection provide answers to certain questions 
or types of questions recurrently asked by the patrons of the library 
In which the staff member works. Due to a lack of training In the 
reference reasoning process, the untrained reference staff member 
supposedly will not structure his experience In such a way as to In- 
crease skill In a step-by-step decision process whereby effective 
access can be gained to a significant portion of a collection of 
Informational documents assumed to be far too large to attempt direct 
acquaintance with but a small portion. Rather than developing a 
pattern of thought wherein, as Hutchins puts It, several steps are 
taken "before a hand Is lifted toward a book" (at least a book 
expected to contain the actual Information sought), the untrained 
reference librarian Is expected typically to turn very early In his 
reference process to a document he hopes will contain the needed 
Information, and Inefficiently to continue to turn to other 
documents until the needed Information Is produced or the range of 
possibilities under his control Is exhausted. 

In brief, then, reference administrators and teachers have 
asserted that all reference questions should be answered by librarians 
with formal library education. This assertion rests on the expect- 
ation that library education and required pre-professional education 
will equip the librarian with knowledge and skill necessary to pro- 
ducing the answers to patrons' questions. Formalizing this assertion 
and expectation somewhat, the following general hypothesis was derived: 

Other things being equal, the professionally trained 
reference librarian will be able to answer a larger 
proportion of Information requests and will do so In 
less time than will the untrained staff member, 
because the trained librarian will have developed 
skill In a reference process that affords him 
effective access to the Informational contents of 
a larger portion of the library's collection. 



General Research Method 



The general research situation that was chosen for this study was 
that of the field study. In which the hypothesis could be studied 
In at least a limited way, while allowing for the discovery and 



exploration of interrelationships among other important factors in 
reference staffing. Given the time limitation of the study, the 
only feasible way to manipulate the independent variable seemed to 
be to compare subjects having formal library education with subjects 
lacking such education. This determined the use of a research 
strategy often called the ^ post facto experiment, wherein an 
attempt is made to trace a measured difference on the dependent 
variable back to a pre-existing difference on the independent 
variable, not directly manipulated by the investigator. 

As mentioned above, not a great deal is known about the 
operation of variables relevant to the performance of reference 
personnel. However, experience and common sense do suggest some 
variables that should be controlled in such an experiment as this. 
Undoubtedly, the speed and accuracy with which a reference question 
can be answered are affected by such factors as size and composition 
of the book collection, physical arrangement of the library, and 
local procedures— factors which always vary from library to library. 

The most feasible way to minimize the effect of these interlibrary 
variations appeared to be to use a pair or pairs of staff members at 
each of several libraries. One member of each pair was to have a 
fifth-year library degree and the other little or no formal library 
training. Other factors possibly or probably relevant to reference 
performance are amount of undergraduate or non- library education and 
amount of experience in reference work. These were to be controlled 
by making sure that the members of each pair were approximately equal 
in these respects. Other possible variables present themselves, e.g., 
age, intelligence, personality, etc. While matching on these might 
well have increased the power of the test of the hypothesis, it was 
feared that attempting it should too severely reduce the number of 
available subjects. 

The experience of the investigator and data from the ALA 
reference survey^^ indicated that medium-sized public libraries often 
employ both professionally trained and untrained .staff to answer 
reference questions. This size and type of institution was therefore 
chosen for the study. To simplify scoring procedures, and because 
a large proportion of reference questions have been found to fall 
into this category, 20 only questions asking for verifiable factual 
information were to be studied. Thus, the general hypothesis was to 
be tested by attention to a considerably more specific proposition, 
i.e., that other things being equal, in a given medium-sized public 
library a reference librarian with a fifth-year library degree will 
be able to answer more reference questions asking for specific factua 
information, and to do so more quickly, than will a reference staff 
member with little or nor formal library school training, because 
the trained librarian will have developed skill in a reference process 
or technique that affords him effective access to a larger amount of 
the information contained in the library's collection. 
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Selection of Cases 



Before cases could be selected for study, it was necessary to 
identify potential pairs of reference staff members. This was 
accomplished by sending in June of 1966 a survey questionnaire con- 
cerning reference staff to all medium-sized public libraries in 
Illinois, Indiana, Iowa, Missouri, and the southern tier of 
counties of Wisconsin. Since the purpose of this survey was merely 
the identification of potential cases and not the gathering of 
detailed information on reference staffi' i, the results will not 
be presented here. Some data from the sui 'cy, as well as more 
detailed information on the definitions and method used, have been 
reported elsewhere. 

The survey population (excluding two libraries which were 
queried but which were judged later to fall outside the defining 
limits) included 69 libraries serving populations of 13,368 to 
184,714; having 51,418 to 341,000 volumes; spending $31,200 to 
$287^00 on salaries; and open 58 to 77 hours per week. The medians 
for the surveyed libraries were 51,393 population served, 93,313 
volumes held, $77,500 salary expenditure, and 69 hours open. 
Sixty-one, or 88 pd^cent, of the libraries returned completed 
questionnaires. The respondents were spread across the entire survey 
population ranges on the defining factors. The medians for the 
responding libraries were 51,397 population served, 89,059 volumes 
held, $78,300 salary expenditures, and 69 hours open per week.^^ 

The questionnaire asked the libraries to indicate the total 
number of reference staff members who gave reference or information 
service to adults for as many as eight hours per week from the 
central or main reference department or collection. Of this total 
they were then asked to indicate the number of staff members holding 
fifth-year library degrees, the number with undergraduate library 
science majors, the number with two years or more of college and 
three or more courses in library science, and the number with two 
or more yea.rs of college but less than three courses in library 
science. For each person indicated in the four education categories 
they were asked to give the number of years of reference experience. 

Originally it had been thought possible to construct pairs with 
various combinations of the four education categories mentioned 
on the questionnaire, while holding library and experience constant 
for each pair. Analysis of the questionnaire returns showed that 
it was advisable to use only the "fifth-year degree" and the "less 
than three library science courses" categories. The plan for the 
study included some attention to differential performance by staff 
members with different amounts of reference experience. Thus, 
pairs of staff members used in the present study were to have varying 
amounts of reference experience. 

On the basis of the survey, 21 libraries were chosen as having 
staff characteristics potentially appropriate for study. Telephone 
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conversatlons and correspondence with the head librarians of these 
institutions reduced the number to twelve* Four of these twelve 
were chosen to pretest the various procedures and instruments for 
the study, and eight were chosen for the actual data- gathering field 
visits. Data from one library was ultimately not usable because 
one staff member was ill at the time of the visit, leaving seven 
libraries with nine pairs of staff members. 

Table 1 shows the characteristics of the libraries studied, 
and Table 2 shows the characteristics of the participants. In 
the library where there were three pairs, the investigator 
decided that one pair set up on the basis of information from the 
administrator did not meet the requirements of the study. However, 
since the untrained member of the pair (F2 from Table 2) was 
fairly well matched with the trained member (El) of another pair, 
it was decided to use the data from this latter librarian again, 
thus creating the pair Fl (El repeated), F2. In all computations 
and tables presented throughout this report the scores for the 
trained members of pairs E and F represent the scores of the one 
trained librarian repeated. It was not possible to match each pair 
precisely on the amount of undergraduate and non-library-science 
education. The amounts held are shown in Table 2, and the possible 
effects of the differences are considered in the sections dealing 
with findings. 



TABLE 1 

LIBRARIES IN WHICH PARTICIPANTS WORKED 



Library 


Population 


Volumes 


Salary 

Expenditure 


Participants 


1 


108,458 


159,504 


$181,000 


Al, A2 


2 


63,715 


131,554 


93,700 


Bl, B2 


3 


33,589 


62,319 


38,500 


Cl, C2 


4 


59,364 


128,503 


168,700 


Dl, D2 


5 


55,719 


94,477 


144 , 100 


El, E2 
Fl, F2 
61, 62 


6 


71,755 


149,813 


108,600 


HI, H2 


7 


47,330 


71,451 


53,000 


11, 12 


Median 


59,364 


128,503 


$108,600 
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lABLE 2 

CHARACTERISTICS OF PARTICIPANTS 



Participant 


Reference 

Experience 

(Years) 


Amount of 
Library 
Education 


Other College 
Education 
(Years) 


Age 


Library 


A1 


1 


Degree 


4 1/4 


46 


1 


A2 


2 


3 Courses 


4 


50 


1 


B1 


2 


Degree 


4 


24 


•^2 


B2 


2 1/2 


None 


4 


31 


2 


Cl 


3 


Degree 


4 


25 


3 


C2 


3 


1 Course 


1 1/2 


32 


3 


D1 


3 1/2 


Degree 


4 1/4 


28 


4 


D2 


4 


None 


4 


40 


4 


El 


8 


Degree 


4 


57 


5 


E2 


7 


None 


2 1/2 


49 


5 


FI 


8 


Degree 


4 


57 


5 


F2 


5 


None 


4 


52 


5 


61 


11 


Degree 


4 1/4 


47 


5 


62 


9 


None 


4 1/4 


51 


5 


HI 


10 


Degree 


4 


44 


6 


H2 


12 


None 


4+ 


58 


6 


11 


19 


Degree 


4 


50 


7 


12 


20 


None 


2+ 


60 


7 
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Measurement of Reference Performance 

There is no precedent in the research literature for procedures 
to measure the effectiveness of the reference performance of staff 
members. Ideally, perhaps the best way to compare the accuracy 
and speed with which staff members are able to answer questions 
would be to observe carefully their performance in answering 
actual questions asked by patrons in the library. Preliminary 
planning included this procedure as a possible approach. Re** 
flection on past reference experience, as well as an early 
exploratory study in a public library reference department, pointed 
out the fact that in a reasonable amount of time one cannot expect 
two staff members to be asked a large enough number of questions 
with sufficient similarity to allow comparison of their performance. 

Construction of the test . — The alternative that seemed to 
involve the least danger of bias was to construct a test set of 
questions composed of reference questions that had actually been 
asked in libraries similar to those studied. The investigator was 
to ask these questions of each staff member studied, under conditions 
as similar as possible to the natural reference situation, and to 
observe and record carefully his performance in answering the 
questions. 

A letter requesting such questions was addressed to 24 head 
reference librarians in Ohio, Michigan, Wisconsin, and Minnesota. 
Notices asking for questions were also placed in professional 
journals. These procedures yielded some 350 questions, from 
which those that asked for verifiable factual (not interpretative 
or judgmental) information were separated. In order to assess the 
possibility and approximate difficulty of answering these questions 
from a public library collection, the investigator attempted to 
answer each of them in a local medium-sized public library. 

In an attempt to increase confidence in the investigator's 
judgment of the difficulty of the questions, the judgment of other 
reference librarians was sought. Forty questions were listed on 
each of two rating forms. The forms, along with letters asking 
librarians to rate the difficulty of the questions, were sent to 
each o£ 47 libraries in the original survey population which had 
responded to the questionnaire but which had not been chosen to be 
studied (see Appendix B). Thirty-seven sets of completed forms 
were returned. In general the judgments of the responding librarians 
supported those of the investigator, particularly on those questions 
finally chosen for the test. 

When it came to actually constructing the set of questions, 
several rather arbitrary decisions had to be made. The study was 
planned with the expectation that at least three hours would be 
spent with each staff member asking the test reference questions. 
Allowing time for a coffee break and for the informal discussion 
of each question, and figuring on from a few seconds to several 
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mlnutes answering time per question, it was anticipated that all 
persons studied should be able to complete just under 20 questions* 
Since it was desirable for each participant to be exposed to 
questions varying on a number of dimensions, it was decided to 
construct a group of from 15 to 20 questions containing as much 
variety as possible. In order to obtain additional data from those 
persons who would be able to complete a larger number of questions, 
another similar but shorter set was made up and used also. 

The initial group of questions contained 17 items. There was 
a fairly even spread on the dimension of difficulty, as judged by 
the investigator and the rating librarians. Most subject fields, 
as categorized by the Dewey Decimal Classification system, were re- 
presented, with concentrations in the social sciences and the area 
of science and technology, in accordance with previous studies on 
reference questions. Questions typical of those arising out of 
school work (a very significant component of adult reference work) 
and of those arising out of more practical situations were included. 
Finally, there were variations among questions with a high probability 
of being answered from strictly reference materials, those probably 
requiring circulating books, and those requiring periodical materials, 
again in keeping with previous studies. ^5 The second group of 
questions in the total set contained 12 items, in general varying 
on the same dimensions. The arrangement or sequence of the questions 
within each group was determined by the desire to place less 
difficult questions throughout the test, to provide encouragement 
and satisfaction, and to avoid any self- instructional features in 
the progression of the test (see Appendix C). 

Scoring .— The scoring procedure that was used on the test 
instrument was to categorize each response to an attempted question 
as either "answered correctly" or "not answered correctly." This 
is an admitted simplification of what happens in library reference 
work and what happened in response to the test questions. However, 
it was felt that a more precise or detailed scoring procedure 
would exceed the reliability and validity of the test instrument. 

For scoring purposes all answers to a single question were 
considered together. For each question the best answer produced by 
any participant was used as a criterion for correctness. This 
"best" was selected for having been based on the most up-to-date 
and authoritative Information. Then each answer judged similar 
enough to this best answer to be of about the same usefulness 
to a patron was scored as "answered correctly." Answers to the 
question under consideration that were untrue or contradictory 
to the correct answer and the failure to provide any answer to an 
attempted question were scored "not answered correctly," as were 
answers that were partially correct but judged more similar in 
usefulness to no answer than to the "best" answer. Each answer 
was also scored in terms of the time taken to produce it. The 
number of questions answered correctly and the time taken to produce 
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the answers were combined in various ways, as will be indicated in 
the sections dealing with findings. 

Validity . — Ejicause of the dearth of previous research in the 
area, hence of objective criteria, and the exploratory nature of 
this study, little of a rigorous or statistically sophisticated 
nature could be done to establish the validity of the measuring 
instrument. Some steps were taken to insure or check some types 
of validity and thus to increase confidence in the general 
presumptive validity of the test. The paragraphs above outline 
a careful attempt (i.e., gathering the questions from actual 
departments similar to those studied, seeking Independent judgements 
on their difficulty, varying the type of questions in accordance 
with experience and research) to insure that the test was 
representative of the universe being measured--a measure of 
content validity. Informal comments regarding the comparison of 
the set of questions to those typically asked of them were 
elicited from participants; these generally increased confidence in 
the validity of the test. 

One outside criterion which the study attempted to use as a 
check on the predictive or concurrent validity of the test was the 
opinion of the supervisors as to the quest ion- answering ability 
of the participants, as obtained in interviews. This attempt was 
only partially successful, since some administrators were hesitant 
to make a distinction. However, usable information from these 
Interviews tended to support the validity of the test. 

Finally, the investigator's judgment of the comparative 
reference performance of the staff members studied was used as a 
criterion po assess the validity of the test. This judgment was 
based on close observation for at least a three-hour period of each 
subject doing actual reference work with patrons. Although for a 
few participants the three hours yielded few questions for 
observation, the comparative judgments reached on the basis of the 
observations agreed with the differences on the test. 

On the whole, the investigator felt that sufficient evidence 
was found to consider the test and scoring procedures adequately 
valid at least to rank the participants and to rank the differences 
among them, thus allowing statistical manipulations appropriate to 
this level of measurement. 

Observation/ interview guide . — It was hypothesized that differences 
in speed and accuracy between reference staff members could be 
related to a difference in their procedure or technique for answering 
questions. Thus, a major portion of the data gathering had to be the 
careful recording of the staff members' behavior while they attempted 
to answer the questions. Since little was known as to what specific 
kinds of behavior might turn out to be relevant to answering 
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efficiency, it was desirable to make as complete a reocrd as possible 
of the participants* behavior and even thoughts if possible. After 
varfous means for mechanically recording the processes were con- 
sidered and rejected, it was decided to construct an observation 
schedule on which to record overt behavior, accompanied by an 
interview guide to attempt to get at relevant thought processes 
(see Appendix D). 

The observation schedule was designed so as to allow recording 
behavior by simply checking appropriate categories, while avoiding 
the necessity for forcing behavior into categories that were too 
restrictive. This latter was accomplished by being as detailed as 
possible in the categories and leaving space for notes on each 
behavior step. Previous studies and texts on reference work gave 
some indication of what types of behavior might be expected, though 
they tended to dwell on what should happen rather than on what 
actually does ha^ppen. Works by Alexander, ® Barton, ^7 and Hutchins 
were particularly helpful. On the interview guide to be used afttir 
each test question was attempted some questions were left rather 
open-ended, to get at thought processes unant icipan ted by the 
investigator; other questions were specific, to gather information 
on particular decisions felt to be relevant to reference technique* 



Administration of the Test 



The actual testing sessions began with the investigator's 
reading the same instructions to each particupant and answering 
any questions he might have (see Appendix E). Then each question 
was asked in turn, and the investigator followed the participant 
about as he attempted to find an answer. As soon as the question 
was stated once by the investigator, a stop-watch was started. As 
the particpant attempted to find the answer, each observable 
step was noted on the observation schedule and each relevant verbal 
comment recorded. As soon as an answer was pointed out, or failure 
admitted, the stop-watch was stopped and the time recorded. The 
search for an answer was not allowed to go beyond 15 minutes, unless 
it appeared that the next step would produce the answer. This was 
done to insure coverage of the entire first group of questions by 
each participant. After each question was attempted, the invest- 
igator asked the participant questions, under the guidance of the 
interview schedule, about the decisions and thoughts involved in the 
finding of the answer. The answers to these questions were recorded 
on a copy of the interview guide provided for each question. It 
was found more useful to deal with the information from this 
observation/ interveiw guide somewhat directly, rather than via a 
formal scoring procedure. Thus, methods of abstracting and com- 
bining data from this guide will be discussed when the findings are 
discussed below. 
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Other Variables 



As was mentioned above, the choice of the general research 
method used in this study was dictated in part by the desire to 
gain information on the operation of variables other than the 
major ones of the hypothesis. Thus, data on several other 
characteristics of the participants and the reference situation 
at each library were collected. This section will discuss 
procedures for collecting this data and for arriving at rankings 
for the different variables. Subsequent chapters will discuss 
findings resulting from these procedures. Two interview guides 
were constructed to aid in gathering this information. One 
guide was used to structure an interview with the supervisor of 
each pair of reference staff members studied. In most cases the 
person interviewed was the head of the reference department or 
the head of adult public services. In cases where the reference 
administrator was a subject for the study, the head librarian was 
interviewed. One of the main purposes of this interview was to 
get the impressions of the supervisors regarding the comparative 
ability of the participants to answer reference questions, as 
mentioned in the section on validation above. Questions regarding 
the library collection were also asked, as well as a question 
concerning in-service training programs for reference staff. Each 
supervisor was also asked about his library's policies and procedures 
regarding the referral of questions from one staff member to 
another, on the assumption that a staff member who had always been 
required to refer difficult questions to another librarian would 
be at a disadvantage in attempting the test questions. None of the 
libraries was found to have policies or procedures worked out formally 
enough to raise this danger. Finally, questions were asked regard- 
ing the library practices on obtaining outside help on questions, 
again on the assumption that this might make a difference in the 
staff members' performance. Again, not enough difference from library 
to library was found to matter, since all professed to use other 
sources to a moderate extent. 

The second interview guide, used in an interview with each 
reference staff member studied, was designed in part to check on 
how well members of each pair were matched on the variables mentioned 
in earlier sections, such as amount of undergraduate and non-library 
education and amount of reference work experience. In addition, it 
was anticipated that the ages of the participants, the major subject 
of non-library education, in-service training and supervision, 
number and content of formal reference courses, non-reference work 
experience, and non-question-answering duties regularly performed 
might be variables worth attention. Accordingly, questions con- 
cerning these were included in the formal interviews. 

In addition to the formal interviews, considerable information 
was gathered from informal unstructured conversations with 
participants and administrators and from observations and note- taking. 
One unanticipated variable on which attention was focused by these 
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discussions was the relative amount of time a staff member actually 
spent at the desk answering questions during a typical week. This 
information was gathered for each participant and is related to 
performance later in this report. Likewise, some of the librarians 
considered the variable of time elapsed since the completion of 
formal education to be as important or more so than that of age, 
so that this is taken into consideration in the findings also. 

Finally, notes were taken on the procedures and physical layout 
of each library relevant to reference work. 

In-service training and supervision . --Although they were 
treated as separate matters on the interview guides, as the study 
proceeded it became clear that in-service training, amount and kind 
of professional supervision, and self-education techniques, as well 
as the unanticipated variable of amount and type of contact with 
non-supervisory colleagues, could more usefully be considered 
together as conctit'«i'^.ing one variable or construct. It was also 
found that the only meaningful differences on these variables 
were among the participants lacking the fifth-year professional 
degree. Findings on the various components of the in-service 
training and supervision construct were considered and the 
untrained participants were ranked according to the relative 
amount of the construct as a whole each was judged to have had. 

For the two untrained participants who had some formal library 
science, this fact was also used in arriving at the rankings. 

Findings on this variable are discussed below on page 31. 

Non- cue St ion- answering duties and other work experience . --Work 
experience other than library reference work was so varied and lack- 
ing in pattern that it did not constitute a meaningful variable to 
be related to performance. Non-question-answering duties regularly 
performed by the participants were handled in two parts. First, 
those duties connected with the selection and handling of reference 
books were separated out for each participant. These consisted 
mainly of responsibilities for helping select reference books for 
purchase and for processing or examining all or some of the reference 
books purchased. These duties were considered as a whole for each 
participant, relative amounts of difference in such duties were 
assessed for each pair, and the pairs were ranked according to the 
sizes of the differences. For example, in one pair neither member 
had any responsibility for helping select reference books and in 
another both members had considerable such responsibility; these 
were judged to have no difference on this factor and were ranked low. 

In other pairs the trained member had substantially more such 
involvement in book selection than did the untrained member, and these 
pairs were ranked high. The findings are discussed on pages 32 and 33. 
The second part of the information concerning duties other than 
answering questions dealt with a wide variety of duties, which 
lacked enough pattern to permit scoring or rankings. They are dealt 
with more directly in relation to reference procedure in Chapter 
III. 
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Differences among libraries . — It was anticipated that a large 
proportion of the differences in reference performance among 
participants, whether trained or untrained, would be due to various 
difference among the libraries in which the participants worked, 
hence the use of matched pairs at each library. As mentioned 
above, notes were taken on such differences to substantiate this 
belief, and as the study progressed the soundness of the belief 
became more and more evident. Examples of factors that differs 
from library to library and that were judged to affect reference 
performance seriously are space problems forcing the separation an 
special shelving of portions of the reference collection, location 
of part of the adult non-fiction circulating collection or magazine 
collection in areas at a distance from the reference desk or card 
catalog, size and content of separate "ready reference" collections 
located near the desk, and policies of the libraries regarding the 
classification of reference- type materials to circulate. 

These differences could be observed and recorded in only a gross 
and sometimes rather subjective way, so that the information obtained 
about them was not considered precise or complete enough to score 
or treat statistically. This means that the only way to account for 
or hold constant these inter-library differences was to keep the 
members of each pair together throughout the analyses. In many 
instances it would be very interesting to see what relationship 
existed between differences on a given variable, e.g., years of 
experience, and differences in performance, among all trained 
librarians, all untrained staff members, or all participants, disre- 
garding the pair to which they belonged. Since the meaning of any 
apparent relationship found in such cross-library analyses would 
]j 0 brought into question by the failure to account for the effect 
of inter-library differences, such analyses are not attempted in 
Chapter II, where performance is related to the independent variables. 
Several of these factors noted at the various libraries will be dis 
cussed throughout Chapter III, In terms of how they were judged to 
relate to the reference technique or procedure of individual 
participants. 

One variable that differed from library to library and on which 
fairly objective data was available was collection size. Originally 
it had been thought that the size of the reference collection might 
have a significant differential effect on the performance of 
trained and untrained staff members. However, it was found that 
the participants moved so freely between the reference collection and 
the adult circulating collection that the size of the adult col- 
lection of the central or headquarters library was considered a 
better measure of the collection of materials with which the in- 
dividual participants had to work. It was this figure that was 
asked for in the interview with the supervisors. Findings concern- 
ing this factor are reported on pages 36 and 37 . 
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Pretesting 



Pretesting of the various data collecting devices and procedures 
was done in several stages. After the instruments were initially 
constructed they were tried with a staff member in a local public 
library. Several major changes were made as a result of this trial. 
The procedure was then repeated in another nearby public library. 

Ths time minor changes were made. Finally, pretest trips as 
possible to the anticipated field visits were made to ^o medium- 
sized public libraries in surburban Chicago. Changes in the 
instruments and procedures made as a result of these visits were 
few, and it was judged that sufficient practice had been obtained 
in timing and recording for the actual field visits to begin. 



Scheduling the Visits 

The visits to the participating libraries were scheduled so 
that the better part of two working days could be spent in each 
library (more, of course, in the library with three pairs). This 
gave time for two half-day sessions with each participant, one in 
which to ask the test questions and one during which to observe 
actual reference work with patrons. The former was often in the 
morning, when the reference department was not too busy and a staff 
member could be spared to give full and complete attention to the 
test. The latter was always during the afternoon or evening, 
when the department was expected to be busiest. In addition, time 
was allowed for interviews, as mentioned above. Finally, as was 
noted above, at each library there was opportunity for informal 
visits, during which relevant and useful information, not anticipated 
at the time the interview schec’ .les were drawn up, was gathered. The 
field visits were made during October and November of 1966. 
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CHAPTER II 



FINDINGS : GENERAL 



The preceding chapter outlined procedures followed in gathering 
the data for this study. This chapter is the first of two that 
will detail the findings resulting from these procedures. Here 
the relationships between the two major variables, professional 
training and quest ion- answering performance, will be discussed, as 
well as the relationship between performance and the several other 
variables mentioned in the previous chapter. The proposed causal 
element, i.e., the technique or procedure used in finding answers, 
will be analyzed in the next chapter* 

Table 1 in Appendix A shows the number of questions attempted, 
answered correctly, and not answered correctly by each participant. 

An X represents a question answered correctly, and an 0 designates 
a question not answered correctly. Dashes have been used to indicate 
questions not attempted. Table 2 in Appendix A lists the number of 
minutes used by each participant on each question attempted. 

Table 3 contains summary data from Tables 1 and 2. Parts of this 
data are repeated in more convenient smaller tables throughout this 

chapter. 

The scores reported throughout this chapter were arrived at 
by taking the arithmetic average between two basic units. The 
first of these consists of questions 1-17 on the test set and the time 
spent attempting to find answers to them. This unit was used be- 
cause it comprised the basic test set and because each participant 
attempted each of these questions in the same order, making it 
useful for comparison. The other unit is the first 100 minutes of 
actual question- answering time and the questions attempted during 
that time. This unit was also used because of its comparability 
among all participants. Each staff member spent at least 100 minutes 
actually seeking answers to questions presented in the same order. 
Since some participants answered more and thus different questions 
in 100 minutes than did others, the scores on this unit may not be 
quite as comparable as that on the first 17 questions, and yet it 
seemed desirable to use these scores since they utilized information 
on a broader range of effort for many participants. Scores on both 
units are presented in Table 3 in Appendix A. It was considered 
that averaging these two units would use information from both, 
while minimizing any bias present in either one alone. 
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Professional Education and Reference Performance 

This section will present the findings on the relationships 
between formal library education and the speed and accuracy with 
which the test reference questions were answered. The former 
variable will be treated as a dichotomy, with the nine staff members 
having fifth-year library degrees (designated as "trained" through- 
out this report) being compared with the nine having little or no 
formal library education (designated "untrained"). The findings 
on the performance variable will be presented in a number of ways, 
which should become clear as the report progresses. 

The basic statistical test used to test the significance of the 
various relationships found is the Wilcoxon matched-pairs signed- 
ranks test.^ The choice of this test was dictated by several 
factors. First, very little is known about the parameters of the 
population from which the study cases were drawn, hence a nonpar- 
ametric test is more appropriate. In addition, the level of measure- 
ment achieved was considered to be that of ranking, ruling out tests 
requiring higher levels of measurement on the variables# Finally, 
the study design used matched pairs, rather than independent samples, 
further limiting the choice of appropriate tests. 

Briefly, the rationale of this test is as follows. The 
independent variable constitutes two treatments, A and B (here 
training and lack thereof), and the research hypothesis predicts an 
effect on the scores representing the dependent variable. Each pair 
has a difference on the dependent variable which can be ranked 
according to the magnitude of the difference, regardless of its 
direction . If treatments A and B are really equivalent, that is if 
the null hypothesis is true, one would expect some of the larger 
differences to favor treatment A and some to favor treatment B. Thus, 
if the ranks favoring treatment A, e.g., positive, were summed, and 
the ranks favoring treatment B, e.g., negative, were summed, the two 
sums should be about equal. But if the sums were very much different, 
one would infer that treatment A differs from treatment B, and thus 
would reject the null hypothesis and hold the research hypothesis 
tenable (assuming the difference was in predicted direction). 

The Wilcoxon test uses the statistic T, which is the smaller 
sum of the like-signed ranks. The ditribution of the statistic T 
has been calculated, and tables or formulas can be used to determine 
critical values for various levels of significance, that is, the 
probability of occurrence of such values for T if the variables were 
not related. The null hypothesis predicts similar sums of the 
negative and positive ranks, producing a large T. A small T would 
indicate a difference in treatments and is what is required to reject 
the null hypothesis in favor of the research hypothesis. In the 
calculation of T where there are pairs^with no difference, the 
procedure recommended by Hays is used. 
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Accuracv, as related to library education . --One of the fears 
underlying the staffing nonti or principle from which the study 
hypothesis was derived is that untrained staff members will more 
often fail to answer reference questions correctly, thus compromising 
a basic function of the library. 3 Counting accuracy alone, the data 
from the participants were examined in terms of the number of 
questions answered correctly among the first seventeen and t e ^ 
number answered correctly of those attempted in the first 100 minutes 
of answering time. The results are shown in Table 3. It will be 
noted that the differences are quite small in most cases. In 
general, the trained members of the pairs appear to have answered 
correctly a larger proportion of the questions attempted. However, 
the table shows that among the three pairs where differences were 
largest, for one pair the direction of the difference was in favor 
of the untrained member. Thus, the T value is so large that the 
null hypothesis of no difference can be rejected only at a level 
considerably above .05, indicating that the apparent difference is 
not statistically significant. 



TABLE 3 

RATIO OF QUESTIONS ANSWERED CORRECTLY TO QUESTIONS 
ATTEMPTED RELATED TO LIBRARY EDUCATION 



PaiiT 


Trained 

Member 


Untrained 

Member 


Difference 


Rank of 
Difference 


Negative 

Ranks 


Of » 

A 


.92 


.77 


.15 


8 




B 


.86 


.86 


.00 


1.5 




c 


.95 


.91 


.04 


3.5 




D 


.88 


.94 


(-).06 


(-)5.5 


5.5 


E 


.88 


.82 


.06 


5.5 




F 


.88 


.98 


(-).io 


(-)7 


7 


G 


.95 


.95 


.00 


(-)1.5 


1.5 


H 


.98 


.94 


.04 


3.5 




I 


.83 


.66 


.17 


9 

T 


f— 1 

II 



To be more specific, some reference administrators would 
undoubtedly say than an incorrect or misleading answer, on which a 
patron might base a wrong decision, is worse than simply failing to 
find an answer to a question. Accordingly, the number of times the 
participants gave incorrect answers was checked. On the first 17 
questions only one incorrect answer each was given by two partic- 
ipants, one trained and one untrained. In 100 minutes of answering 
time one question was answered incorrectly by each of four trained 
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staff members and one each by three untrained staff members. A 
comparable small difference was found regarding partially correct 
answers that were scored as "not answered correctly." At the 
level of the factual questions represented on the test instrument, 
then, the fear of a significant difference in accuracy between 
trained and untrained reference staff members finds little support 
from this study. 

Speed, as related to library education . --To compare the speed 
with which the trained and untrained participants dealt with 
questions asked them, regardless of the outcome of the action, the 
test results were scored in terms of the time taken per question 
attempted. The findings are shown in Table 4. It should be noted 
that higher scores on this table indicate slower speeds, since 
the figures represent average minutes required to work on a 
question. In all cases but one the trained members were faster. 

The probability of this happening if the variables were not related 
is small, and the null hypothesis can be rejected at the .025 signi- 
ficance level. 



TABLE 4 

MINUTES TAKEN PER QUESTION ATTEMPTED 
RELATED TO LIBRARY EDUCATION 



Pair 


Trained 

Member 


Untrained 

Member 


Difference 


Rank of 
Difference 


Positive 

Ranks 


A 


5.23 


5.42 


.19 


1 




B 


4.98 


5.50 


.52 


4 




C 


3.14 


3.87 


.73 


6 




D 


5.28 


4.66 


(+).62 


(+)5 


5 


E 


4.14 


6.49 


2.35 


9 




F 


4.14 


4.36 


.22 


2 




6 


3.86 


5,25 


1.39 


8 




H 


3.55 


5.91 


1.36 


7 




I 


5.02 


5.26 


.24 


3 


T = 5 



Again, more specifically, some administrators fear that reference 
staff members lacking formal training will not know the information 
sources of the library well enough to know when to stop on a question 
to which an answer cannot be readily found and will thus unduly 
waste the time of the patron. To test this assumption the results 
of the test were examined to compare how many times the trained and 
untrained participants failed to decide to give up on an unanswered test 
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quest ion before the rather arbitrary 15-mlnute time limit mentioned 
above (assuming that those who gave up with no answer would have 
in the real situation referred the patron to another staff member 
or asked the patron to call back later). There was some difference 
found, though assessing its significance is difficult. Two un- 
trained staff members would have exceeded the 15-minutes limit on 
two questions each and another on one. Among the trained librarians, 
only one participant failed to answer or give up on only one question 
within the time limit. This point will be considered again in the 
next section. 

Efficiency, as related to library education . — Combining accuracy 
and speed, a score can be arrived at which might be called reference 
performance efficiency. That is, by combining the total time spent 
attempting questions and the number of questions answered correctly 
during that time, the participants can be scored with regard to the 
efficiency of their performance. Table 5 shows the results of this 
analysis. Again, it should be noted that a higher score represents 
lower efficiency. In all cases but two the trained members of 
the pairs were more efficient than were the untrained members, and 
these two cases had differences among the smallest. The probability 
of obtaining such a small T value would be very small if the 
variables were unrelated, and the null hypothesis can be rejected 
at the .025 level, thus adding to the credibility of the study 
hypothesis. 

The one study, cited in the literature survey above, which 
concluded that staff members lacking formal library education 
should be able to answer a large portion of public library 
reference questions, based this conclusion on the assumption that 
such staff members would be able to refer questions for which , 

they could not find answers quickly to librarians with more education. 



TABLE 5 

MINUTES TAKEN PER CORRECT ANSWER 
RELATED TO LIBRARY EDUCATION 



Trained Untrained Rank of Positive 

Pair Member Member Difference Difference Ranks 



A 


5.70 


7.03 


1.33 


5 


B 


5.75 


6.42 


.67 


2 


C 


3.28 


4.24 


.96 


3 


D 


5.94 


4.92 


(+)1.02 


(+)4 


E 


4.70 


7.94 


3.24 


9 


F 


4.70 


4.44 


(+) .26 


(+)1 


6 


4.07 


5.56 


1.49 


6 


H 


3.62 


6.28 


2.66 


8 


I 


6.02 


7.99 


1.97 


7 



1 
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In fact, as mentioned above, it is considered good reference ^ 
practive for any staff member not to waste his own or the patron s 
time while searching an unduly long time for an answer to a 
question, but rather either to refer the question to another 
staff member or to take the patron's name, in order to contact 
him later with the results of an extended search. Thus, it might 
be of interest to see the comparison between trained and un- 
trained staff members, if the decision to give up on a question 
(assuming this implies a referral or "call back ) is considered 
a correct course of action. 

Table 6 shows the findings on this point. As expected, 
many of the staff members show an improvement in their efficiency 
score (again, indicated by a lower figure). The improvement is 
more general among the trained staff members, indicating that 
they decided to "refer" relatively more often. With this scoring, 
in only one pair did the untrained member perform more efficiently 
than the trained member, and the difference in this pair 
among the smallest. Thus the null hypothesis can be rejected at 

well below .025. 



TABLE 6 

MINUTES TAKEN PER CORRECT ANSWER OR REFERRAL 
RELATED TO LIBRARY EDUCATION 



Trained Untrained 

Pair Member Member 



Rank of Positive 

Difference Difference Ranks 



A 

B 

C 

D 

E 

F 

6 

H 

I 



5.23 

5.42 

3.20 

5.60 

4.44 

4.44 

4.07 

3.55 

5.32 



6.12 


.89 


6.42 


1.00 


3.94 


.74 


4.92 


(+).68 


7.94 


3.50 


4.44 


.00 


5.25 


1.18 


6.28 


2.73 


6.38 


1.06 



4 

5 
3 

(+)2 2 
9 
1 

7 

8 

6 



It should be noted that the decision to give up on a question 
could be merely an indication of a lack of tenacity, which could 
work to a patron's disadvantage as well as to his advantage, parti- 
cularly if the staff member were on duty alone when the answer was 
needed. In any case, whether the preferred scoring interpretation 
is to count a decision to give up as "not answered correctly or to 
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score such a decision as a correct course of action, there is 
reason for accepting the hypothesis that professional education 
makes a significant difference in reference performance efficiency. 



Other Variables Related to Performance 

This section will consider relationships between reference 
performance and several variables other than formal library 
education. The ways in which these different variables were 
measured andscored are discussed in the preceding chapter on 
pages 14 to 16. The measure used to indicate performance will be 
efficiency scores presented rn Table 5 above. Thus, rt should be 
remembered that a higher score represents lower efficiency. Two 
approaches will be used in examining relationships between variables. 
In some cases the factor under consideration will be treated as a 
dichotomy, and each pair will be split according to which member 
has more or less of the variable, rather than according to being 
trained or untrained. Then this dichotomy will be related to 
performance and the relation tested by the Wilcoxon test used in 
the last section. The second type of analysis will be to relate 
rankings achieved by the pairs on variables to relative amounts of 
difference in performance. That attempts will be 

made to see if high or low rankings for pairs on different factors 
are related to larger or smaller differences in the performance of 
the same pairs. 

There are two main reasons for discussing these other variables. 
First, since this study was done as a field study, it was not 
possible to control all relevant variables, and thus other hypotheses 
built on other variables might be tenable alternatives to the study 
hypothesis. If it can be shown that splitting pairs according to 
the members* having more or less of these alternative independent 
variables produces no significant relationship to performance, 
then these alternative hypotheses will lose credibility , and the 
credibility of the main hypothesis will be strengthened. Likewise, 
if it can be shown that larger differences on a given variable 
between members of individual pairs are not strongly related to 
larger differences in reference performance between the members of 
the same pairs, the given variable will be less likely to be con- 
sidered as a tenable alternative independent variable to professional 
education. Secondly, even if a variable does not present itself 
as figuring in an alternative hypothesis, it might be of value to 
gain information on its operation in the reference situation. For 
example, in a practical situation it should be of interest to know 
which variables were found in this study to be related to the amount 
of spread in efficiency between trained and untrained staff members 
and how they were related. These relationships, then, will be 
discussed also. 

Age as a variable .— Contrary to expectations based on the 
investigator * s experience, in seven of the nine pairs studied the 
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untrained staff member turned out to be older than the trained 
participant (see Table 2, p. 9). It could thus be hypothesized 
that being younger is significantly related to higher reference 
efficiency, as an alternative to the study hypothesis. To test 
this, the pairs were split on the basis of being older or 
younger and their performance studied. Table 7 shows the results. 
In two of the nine pairs the older participant performed more 
efficiently than the younger, and one of them had the larpst 
difference in performance. The T value derived was so large 
that the null hypothesis could be rejected only at a significance 
level in excess of .10, thus weakening the credibility of age as 
an alternative independent variable to professional education. 



TABLE 7 

MINUTES TAKEN PER CORRECT ANSWER 
RELATED TO AGE 



Fair 


Older 

Member 


Younger 

Member 


Difference 


Rank of 
Difference 


Negative 

Ranks 


A 


7.03 


5.70 


1.33 


5 




B 


6.42 


5.75 


.67 


2 




c 


4.24 


3.28 


.96 


3 




D 


4.92 


5.94 


C-)1.02 


(-)4 


4 


E 


4.70 


7.94 


(-)3.24 


(-)9 


9 


F 


4.70 


4.44 


.26 


1 




G 


5.56 


4.07 


1.49 


6 




H 


6.28 


3.62 


2.66 


8 




I 


7.99 


6.02 


1.97 


7 


T * 13 




Time 


elapsed since 


formal education as a variable.— One 



lacuor aavancea oy auuic wj. ^ 

differences in performance was the difference in amount of time 
that had passed since they were actually in school. These persons 
considered that the formal education situation, regardless of 
subject studied, enhances of forces the development of certain 
mental capacities, e.g., speed and flexibility, which help greatly 
in reference work and which decay over time, giving an advantage 
in reference work to the person who has been out of school the 
shorter time. The amount of time since schooling ended was treated 
as a variable and was related to reference efficiency, much as was 
age. Table 8 shows the years elapsed since the last formal 
education for each participant. Table 9 shows the results of 
dichotomizing time elapsed since schooling and relating it to 



t 



-29- 



TABLE 8 

YEARS ELAPSED SINCE END OF 
FOBNAL SCHOOLING 





Untrained 


Trained 






Pair 


Member 


Member 


Difference 


A 


0 




1 




•1 


B 


10 




2 




8 


C 


13 




3 


10 


D 


18 




5 


13 


E 


21 




15 




6 


F 


30 




15 


15 


G 


29 




16 


13 


H 


35 




20 


15 


I 


41 




19 


22 


performance. In three 


: cases staff members away : 


Erom school longer 


scored 


better (indicated by a lower : 


figure) than 


did the other 


members 


of the pairs. 


thus the T values obtained 


were too large 


to indicate a significant difference 


in performance related to 


time away from the formal school situation. 










TABLE 


9 








MINUTES TAKEN PER 


CORRECT ANSWER 






RELATED TO YEARS AWAY FROM SCHOOL 






Member Out 


Member Out 










of School 


of School 




Rank of 


Negative 


Pair 


More Years 


Fewer Years 


Difference 


Difference 


Ranks 


A 


5.70 


7.03 


(-)1.33 


(-)5 


5 


B 


6.42 


5.75 


.67 


2 




C 


4.24 


3.28 


.96 


3 




D 


4.92 


5.94 


(-)1.02 


(-)4 


4 


E 


7.94 


4.70 


3.24 


9 




F 


4.44 


4.70 


(-) .26 


(-)l 


1 


G 


5.56 


4.07 


1.49 


6 




H 


6.28 


3.62 


2.66 


8 




I 


7.99 


6.02 


1.97 


7 





T = 10 
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Reference experience as a variable .--Since amount of previous 
reference experience was held constant as much as possible for each 
pair, experience does not present itself as an alternative in- 
dependent variable to professional education. However, an ancillary 
hypothesis of the study was that as reference experience of the 
pairs increased, the amount of difference in performance between 
the trained and untrained members would decrease, because on-the- 
job learning gained by the untrained staff members would enable 
them to "catch up" with the trained librarians in reference ability 
at the level of the factual questions used for this study. To 
check this hypothesis a rank order correlation between the amount 
of difference in performance for each pair was calculated using 
the Spearman rank order correlation coefficient (rg). The results 
are shown in Table 10. It will be noted that here and throughout 
the rest of the report the rankings used in calculating the rank 
order correlation are different from those used in the Wilcoxon 
test. This is because in the Wilcoxon test the absolute value of 
the difference is used to determine the ranks, disregarding the 
direction or algebraic sign. In the Spearman rank order correla- 
tion the algebraic value is taken into consideration and the ranks 
assigned accordingly. In the tables for correlation, "d" indicates 
the differences in rank and "d^" indicates the differences squared. 



TABLE 10 

RELATIONSHIP BETWEEN YEARS OF EXPERIENCE 
AND DIFFERENCE IN PERFORMANCE 





Years of 


Differences in 


Rank 




2 


Pair 


Experience 


Performance 


Exp. 


Perf . 


d 


d 


A 


1.50 


1.33 


1 


5 


-4 


16 


B 


2.25 


.67 


2 


3 


-1 


1 


c 


3.00 


.96 


3 


4 


-1 


1 


D 


3.75 


-1.02 


4 


1 


3 


9 


E 


7.50 


3.24 


6 


9 


-3 


9 


F 


6.50 


- .26 


5 


2 


3 


9 


G 


10.00 


1.49 


7 


6 


1 


1 


H 


11.00 


2.66 


8 


8 


0 


0 


I 


19.50 


1.97 


9 


7 


2 

o j2 


4 

- 50 














* .583) 



The hypothesis mentioned above would predict that as years 
of experience increased, amount of difference in performance would 
decrease, or a negative correlation. A positive correlation was 
found, indicating that the data from this study give no support to 
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the hypothesis. This hypothesis, of course, rests on the assump- 
tion that the quality and amount of on-the-job learning would be 
the same for all untrained participants over a given amount of 
time. As will be seen in the following section, this was not 
the case in this study, explaining to some extent the negative 
results. Also, experience was fairly closely related to other 
intervening variables in this study. 

In-service training and supervision as a variable . --As was 
mentioned in Chapter I, this variable is really a construct 
composed of several factors. Since this was found to be a mean- 
ingful variable only for the untrained members of the pairs, it 
is not an appropriate factor to figure in an alternative 
hypothesis to the one which guided the study. However, it should 
be interesting to see if an increase in the relative amount of 
in-service training and supervision is related to the relative 
amount of difference in performance among pairs. Table 11 shows 
the correlation between the ranks assigned to the untrained 
members of each pair, with regard to the construct, and the rank 
of the difference in performance. As might be predicted, a 
negative correlation was found, meaning that as amount of op- 
portunity for on-the-job learning by untrained staff members went 
up, the efficiency of the untrained staff members approached that 
of the trained members of the same pairs. 



TABLE 11 

RELATIONSHIP BETWEEN IN-SERVICE TRAINING 
AND DIFFERENCE IN PERFORMANCE 



Pair 


In-Service 

Training 


Rank 

Difference in 
Performance 


d 


d2 


A 


9 


5 


4 


16 


B 


7 


3 


4 


16 


C 


5.5 


4 


1.5 


2.25 


D 


8 


1 


7 


49 


E 


1.5 


9 


-7.5 


56.25 


F 


4 


2 


2 


4 


G 


3 


6 


-3 


9 


H 


1.5 


8 


-6.5 


42.25 


I 


5.5 


7 


-1.5 


2.25 








Cd^ = 


197.00 










-.642) 
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Duties regularly performed other than answering questions c — 
One hypothesis that can be advanced to predict or account for the 
trained librarian's being more proficient at answering reference 
questions is that the duties and responsibilities assinged to 
the trained librarian by his supervisors, and the learning 
provided thereby, better equip him to answer reference questions 
than those duties assigned to untrained staff members. When 
the information gathered to test this hypothesis was examined, 
the duties that appeared to be the most likely to have any such 
effect, as well as the only group of duties showing enough 
pattern to score and treat statistically, were those connected 
with selecting and handling or examining new reference materials. 
Other specific duties regularly performed will be related to 
reference procedure in the next chapter. 

Table 12 gives the rankings arrived at on examination of the 
data bearing on these duties. Reference to one of the previous 
tables that contain the efficiency scores will show that in 
every pair where the trained participant had a larger amount of 
responsibility for selecting and handling reference materials 
the professional scored better in performance, and in the 
one pair where the untrained participant worked more closely 
with selecting adult reference materials (in this pair the 
trained member worked only part-time) the untrained member 
performed more efficiently. In addition, relatively high cor- 
relation coefficients were found (Table 13) between ranks of 
the difference the members of each pair were judged to have 



TABLE 12 

DUTIES INVOLVING REFERENCE BOOiCS 
ASSIGNED TO PARTICIPANTS 



Pair 


Trained 

Member 


Untrained 

Member 


Rank of 
Difference 


A 


(Little or not 


difference) 


3 


B 


More 


Fewer 


5 


C 


More 


Fewer 


6 


D 


Fewer 


More 


1 


E 


More 


Fewer 


8 


F 


(Little or no 


difference) 


3 


G 


(Little or no 


difference) 


3 


H 


More 


Fewer 


7 


I 


More 


Fewer 


9 



on this variable and the ranks of the difference in their performance, 
indicating that the more widely they were separated in involvement 
with selecting and handling reference materials the more widely they 
were separated in reference performance. 
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TABLE 13 

RELATIONSHIP BETWEEN DIFFERENCES IN REFERENCE 
BOOK DUTIES AND DIFFERENCES IN PERFORMANCE 



Pair 


Difference 
In Duties 


Rank 

Difference In 
Performance 


d 




d2 


A 


3 


5 


-2 




4 


B 


5 


3 


2 




4 


C 


6 


4 


2 




4 


D 


1 


1 


0 




0 


E 


8 


9 


-1 




1 


F 


3 


2 


1 




1 


6 


3 


6 


-3 




9 


H 


7 


8 


-1 




1 


I 


9 


7 


2 




4 










Ed^ = 


* 28 












= .767) 



This hypothesis, which receives support here. Is a logical one. 

One would expect the Increased familiarity with information sources, 
gained through procedures Involved In selecting the materials and 
handling them after receipt by the library, would have a beneficial 
effect on finding answers to questions In these same sources. On 
the other hand, accepting it as an alternative to the study hypothesis 
raises certain problems. First, It falls to account for substantial 
differences in performance between the members of two of the three 
pairs judged to have little difference in regard to responsibilities 
for selecting and handling reference materials. Also, the direction 
of any causal relationship between the present two variables Is 
difficult to judge. It seems possible that those untrained staff 
members who had responsibilities for book selection similar to their 
trained colleagues had these duties assigned to them because they 
were already similar in reference ability, not vice versa. Involvement 
with reference book selection and processing is probably one var- 
iable among others that contribute to the differential ability of 
reference staff members to answer questions efficiently. 

Amount of time spent answering question s. — Several participants 
in this study mentioned that they considered the actual number of 
hours per week a staff member spent at the desk answering reference 
questions Important to reference performance. They seemed to think 
that spending more time at the desk, and thus presumably answering 
more questions, helped a staff member to be able to decide more 
quickly on appropriate answering sources and to find those chosen 
more quickly. The majority of pairs studied here had little or 
no difference in the amount of time per week the members were 
assigned to the reference desk, as can be seen from Table 14, so 
that little could be done to test this belief. 



-34- 



TABLE 14 

HOURS PER WEEK ASSIGNED TO ANSWERING 
REFERENCE QUESTIONS 



Trained 

Pair Member 



Untrained 

Member 



Difference 



A 

B 

C 

D 

E 

F 

G 

H 

I 



26 


25 


1 


40 


40 


0 


8+ 


15 


-7 


8 


25 


-17 


12-15 


10 


2-5 


12-15 


12-15 


0 


30 


18-20 


10-12 


12-15 


12-15 


0 


15 


15 


0 



It can be noted from Table 14 and one of the earlier tables 
which shows the efficiency scores that in the library where three 
pairs were studied (E, F, and G) the participant with the most 
hours per week assigned to answering reference questions did have 
the highest efficiency, and the participant with the fewest 
hours per week had the lowest efficiency. However, other ranks 
do not correlate so neatly, nor do the magnitudes of the differences 
in time assigned to the desk and the differences in performance 
correlate. Also, while one pair in which the untrained member 
worked substantially more hours per week (D) was the same pair 
where the untrained member performed considerably more efficiently, 
the same relationship does not hold for pair C. In short, while 
the belief of the participants who raised the question of re- 
lationship between amount of time spent answering questions and 
level of efficiency might be tenable, and even though there is 
some apparent justification for it in the test scores, support 
provided by the present study is not substantial. 

Formal reference training as a variable . --One factor which 
might be expected to affect the amount of difference between 
the reference ability of a trained staff member and an untrained 
one is the amount and type of formal reference education the 
trained librarian has had. The information from the interviews 
concerning this factor is presented in Table 15. In the 
column headed "Type and Emphasis of Courses" are shown the 
results of a question asking about the emphasis and content of 
the courses. For each librarian there is an indication of whether 
or not the library school attended was accredited by the American 
Library Association, followed by an indication of whether or not 
the courses had an emphasis on a particular type of library. 
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Next, the degree to which the technique or procedure for 
answering reference questions was emphasized in the courses is 
noted. This is to be understood as relative to the typical 
emphasis on titles and characteristics of particular reference 
books. Finally, there is an indication of whether or not 
practice questions were used as a teaching technique in t e 

courses. 



TABLE 15 

NUMBER AND TYPE OF FORMAL 
REFERENCE COURSES 



i 



t 



Trained 
Participant 



Number of 
Courses 



Type of Emphasis of Courses 



1 Unaccredited program; school library 

emphasis; little or no attention 
to technique. 



B1 3 

Cl 3 



D1 



2 



El-Fl Parts of 5 
literature 
and bibliog. 
courses 

G1 2 



HI 3 

II 2 



Accredited program; broad emphasis; 
some attention to technique; prac- 
tice questions used. 

Accredited program, broad emphasis; 
considerable attention to technique; 
practice questions used. 

Unaccredited program; school library 
emphasis; slight attention to 
technique . 

Accredited program; broad emphasis; 
some attention to technique, but 
more on materials. 



Accredited program; broad emphasis; 
considerable attention to types of 
tools and techniques; practice 
questions asked. 

Accredited program; broad emphasis; 
some attention to technique; practice 
questions used. 



Accredited program; broad emphasis; 
slight attention to technique; 
practice questions used. 
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Except for the two participants who took their professional 
degrees from school library programs, the differences on this 
factor are not great and would be hard to score or deal with 
objectively. It is tempting to look at the high scores on time 
required per correct answer, indicating low efficiency, for these two 
librarians and to conclude that the amount and quality of their 
reference education hampered their ability. However, this still 
involves rather risky cross- library comparisons and does not account 
for other factors such as amount of experience (low in both cases), 
etc. A more legitimate comparison would be to look at the relative 
size of the difference in performance for the two pairs involving 
these participants and for the other pairs. As the study progressed 
it did appear to the investigator that these two participants who had 
their library education from essentially school library programs did 
indeed have less advantage in performance over their untrained mates. 
In fact, some specific instances where there seemed to be a connection 
between reference course content and technique or procedure will be 
discussed in the following chapter. However, when all the questions 
were scored and analyzed it was seen that while one of these 
trained participants was exceeded in efficiency by the untrained 
member of the pair, the other such trained member performed 
considerably above the untrained member of that pair. Thus some- 
thing less than conclusive evidence was found for the effect of 
the nature of reference education on reference performance. 

Collection size as a variable .— If the ability of untrained 
staff members to answer reference questions were in fact based on 
memory and direct acquaintance with as many information sources as 
possible, while the trained librarian relies on a more logical 
or indirect access to sources, then one might predict than any 
advantage in performance enjoyed by the trained librarian would 
lessen as the size of the available collection of sources decreases. 

To test this prediction the sizes of the central adult collections 
were correlated with the sizes of the differences in performance 
between the trained and untrained members of the pairs. 

Table 16 shows the size of the adult reference collection and 
of the central adult circulating collection for each library 
studied. While data on the number of titles in the reference 
collections were gathered and are presented here for any value or 
interest they might have, because of differing policies from 
library to library regarding the composition of the reference 
collection, the size of the central adult collection was judged 
a better indicator of the range of materials available for answer- 
ing questions. It will be noted that while there are only seven 
libraries, ranks were assigned on the basis of nine, since three 
pairs worked at library number 3. 
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TABLE 16 

SIZE OF REFERENCE AND CENTRAL ADULT 
CIRCULATING COLLECTIONS 



Pair 


Library 


Volumes in 
Central 
Adult 
Collection 


Rank 


Titles in 
Adult 
Reference 
Collection 


A 


1 


87,600 


8 


2,000 


B 


2 


75,000 


7 


2,100 


c 


3 


38,000 


2 


700 


D 


4 


106,700 


9 


4,300 


E 


5 


65,100 


5 


3,800 


F 


5 


65,100 


5 


3,800 


G 


5 


65,100 


5 


3,800 


H 


6 


35,800 


1 


1,500 


I 


7 


38,500 


3 


2,500 



Table 17 shows that a negative correlation was found between 
rank of collection size and rank of the size of the performance 
difference. This indicates that, contrary to prediction, as col- 
lection size of the libraries studied increased, the differences 
in performance between the trained and untrained participants tended 
to decrease. This, of course, has implications to the reference 
procedure or technique of the staff members and will be discussed 
in more detail in the following chapter (see pages 58 and 59). 

Non-librarv education . --As can be seen from Table 2, page 9, 
in all pairs but C, E, and I the amount of education other than 
library science was quite comparable for each pair. The major 
subjects studied in non- library education were too lacking in 
pattern of variability to relate meaningfully to performance 
efficiency. Seven of the eighteen participants majored in English, 
three in history, three in general liberal arts programs, two in 
secondary social studies education, and one each in biology, home 
economics, and elementary education. Some attempt was made to relate 
subject of non- library education specialization to reference 
technique or answering success on various specific questions, but 
no relationship emerged to be reported. 

The pairs in which the trained members had considerably 
more non- library education than the untrained members ranked 
relatively high on difference in performance, meaning that it might 
be hypothesized that amount of non- library education is crucial to 
performance difference. However, in this study such a hypothesis 
accounts for relatively little of the difference found between 
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TABLE 17 

RELATIONSHIP BETWEEN SIZE OF ADULT COLLECTION AND 
DIFFERENCE IN PERFORMANCE 



Pair 


Collection 

Size 


Difference in 
Performance 


d 




A 


8 


5 


3 


9 


B 


7 


3 


4 


16 


C 


2 


4 


-2 


4 


D 


9 


1 


8 


64 


E 


5 


9 


-4 


16 


F 


5 


2 


3 


9 


G 


5 


6 


-1 


1 


H 


1 


8 


-7 


49 


I 


3 


7 


-4 


2 JA. 

£d^ = 184 
(r =-.533) 



members of pairs, since it says nothing of the six other pairs. 

Also, these three pairs had high rankings on variables which were 
related to performance and which did take all pairs into account. 
Finally, in studying differences in the way answers were sought as 
an explanation for differences in efficiency, none of the differences 
in technique could bt: traced to anything judged likely to have been 
based on non- library education. 



Summary 

This chapter has discussed the findings on the relationships 
between the reference performance of the nine pairs of participants 
and several other variables. There was some apparent relationship 
between having a professional library degree and answering the 
test questions accurately, but it was not statistically significant. 
Those participants with library training did handle the questions 
more quickly. Combining accuracy and speed, it was found that 
efficiency in reference performance was significantly related to 
having formal library education, as predicted by the study hypothesis. 

Several variables were examined as potential alternative 
independent variables to professional education in accounting for 
the observed differences in reference efficiency. Age and time 
elapsed since the end of formal education were found not to bear any 
important relationship to differences in efficiency between members 
of the pairs. The amount of time per week actually spent answering 
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reference questions and the amount of non-library education were 
related to differential performance for some pairs, but were 
judged not to receive conclusive support as alternative independent 
variables to library education. Finally, the prediction of a positive 
relationship between the relative amount of involvement with the 
selection and handling of reference materials and level of efficiency 
was supported, though the relationship was judged as contributory 
rather than crucial. 

The size of the difference between trained and untrained members 
of a pair was considered of interest, since, for example, being able 
to predict the conditions under which a smaller difference would 
occur has practical implications. Thus, ranks on several variables 
were correlated with the magnitudes of the difference in the per- 
formance of the trained and untrained members of the pairs. Higher 
rankings on years of reference experience and difference in involve' 
ment with reference book selection were associated with larger dif~ 
ferences in performance. High rankings on the variables of ^ount 
of in-service training for the untrained members and collection 
size were associated with smaller differences in performance between 
the trained and untrained members of the pairs. 

The following chapter will examine the differences in the 
procedure used by the participants in searching for answers to the 
test questions, as these differences relate to accuracy and speed. 

As judgments are advanced regarding explanation for these differences 
in procedure related to difference in efficiency, it is hoped that 
the specific contributions to reference performance by the variables 
discussed above can be better understood. 
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CHAPTER III 



FINDINGS : REFERENCE PROCEDURE 



Chapter II presented findings on the relationships between 
several independent variables, particularly professional training, 
and reference performance. This chapter will deal with findings 
concerning the hypothesized causal factor, the procedure used to 
find the answers to reference questions. It is intended that by 
examining the latter in this chapter a clearer understanding of 
the nature of the relationships found to exist in the previous 
chapter can be obtained. 

The first section will duscuss the general procedure followed 
by the participants in searching for answers to the test questions. 
The following section will deal with differences in answering 
procedure that were judged crucial to differential efficiency. 

Many of the decisions involved in the data tabulations sup- 
porting the findings in this chapter were necessarily of a 
judgmental and perhaps subjective nature. In addition, most of 
the indications reported here rest on small frequencies of 
occurrence, not admitting of statistical testing. For these 
reasons, this chapter is presented in a somewhat informal way, 
with much of the numerical detail omitted. Thus, this part of 
the report represents the most severe condensation of the 
dissertation mentioned in the preface, where much more detail 
can be found. 



General Procedure Used By Participants 

The causal element of the study hypothesis was based on im- 
plications from the literature of reference instruction and 
administration to the effect that the professionally trained 
reference staff member would have effective access to a greater 
portion of the informational content of a given library collection 
than would the untrained staff member. It was postulated that 
this more effective access would rest on a more logical step-by- 
step procedure for choosing answering sources, involving 
categorization and classification of the question, use of the 
bibliographic and indexing apparatus of the library, and pursuit 
of an answering strategy based on characteristics of the question. 
On the other hand, the untrained staff member, it was supposed, 
would rely on a procedure more akin to a simple stimulus-response 
reaction, with the question providing the stimulus and the response 
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belng recourse to one source after another, as dictated by 
experience and memory, until an answer was found or the possibilities 

exhausted. 

This hypothesis would predict that the trained librarian would 
consult fewer sources as potential answering tools than would the 
untrained staff member. To test this prediction, the number of 
times each participant consulted a source with the expectation 
of finding the answer was recorded, excluding sources consulted 
for citations or leads to answering tools. Table 4 in Appendix A 
shows the results question by question. Table 18 below shows the 
results of a Wilcoxon test of the relationship between professional 
training and the number of tools used on questions 1-17 . Contrary 



TABLE 18 

NUMBER OF ANSWERING SOURCES USED BY TRAINED 
AND UNTRAINED PARTICIPANTS 



Pair 


Trained 

Member 


Untrained 

Member 


Difference 


Rank of 
Difference 


Negative 

Ranks 


A 


40 


39 


1 


2 




B 


37 


47 


(-)IO 


(-)5 


5 


c 


45 


26 


19 


9 




D 


39 


50 


(-)ll 


(-)6 


6 


E 


52 


49 


3 


4 




F 


36 


49 


(-)13 


(-)7 


7 


G 


37 


39 


(-) 2 


(-)3 


3 


H 


49 


33 


16 


8 




I 


43 


43 


0 


1 


T = 21 



to prediction, in four of the nine pairs the untrained member used 
fewer tools than the trained member, producing a T that is much too 
large to indicate a significant relationship. Close examination of 
the observation schedules showed that it was much to simple to say 
that the efficient participants used a complicated rational answering 
procedure, while the less efficient participants used a simple 
procedure based on memory. In very general terms, the procedures used 
to locate answers to the test questions by trained and untrained 
participants were similar. Both the fomally trained librarians and 
the staff members lacking training classified and categorized at 
least most of the questions; both groups used the bibliographic and 
indexing resources of the libraries; and both groups seemed to rely 
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more on rather ad hoc decisions, made as they proceeded through 
the searches, than on previously worked out strategies. 

The chart on pages 44 to 46 represents an attempt to present 
a generalized version of the procedure used by the study Parti- 
cipants to find answers to the test questions. Tins flow chart 
uses the conventional shapes to symbolize parts of the procedure. 

The diamond shaped box has been used for "yes-no decisions 
questions, the answers to which determine the subsequent directi 
of activity, as indicated by the arrows. The squares represent 
operations performed. The small pointed box indicates continuation 
of the chart on another sheet at the designated point. While it can 
be said that this is a fairly accurate representation of the 
procedure followed, almost every question and every participant 
produced some variations on the general pattern. 

This chart was constructed by the following method. The 
observation schedules and the accompanying interview guides used 
to record the behavior and thoughts of the participants as they 
attempted to find answers to the questions were examined carefuUy. 
Those for the trained participants were examined as a group, and 
a generalized model for the procedures followed was abstracted. 
Likewise, the schedules for the untrained participants were 
studied and a general procedure abstracted. On comparison, the 
two general moLls were quite similar, and they were combined into 
the one flow chart presented on pages 44 to 46. Some of the 
decisions represented on the chart were not reported by the parti- 
cipants and were probably made at the subconscious level. However, 
from observed actins or reported conscious mental processes these 
decisions could be inferred. Also, many of the decisions were 
undoubtedly not of the simple "yes-no' nature represented 
chart, but rather involved a complex set of alternatives. » 

for simplicity and clarity of presentation these complex decisions 
were broken down into their compontent "yes-no' parts for charting. 
Each of these basic decisions and the subroutines growing out of them 
will be discussed in turn below. 



The first three basic decisions (A, B, C) represent rather ^ 
quick responses to the question itself and its wording. Subroutine 
A was used when important words in the question were unfamiliar to 
the participants, when the question did not seem to contain enoug 
information to indicate a direction of search, or when the parti- 
cipant judged that the search for the answer might be speeded by 
added information about the question. The most common immediate 
response to such a situation was for the participant to ask 
the "patron" if he could provide the needed information. If tne 
"patron" could not add enough information, the participant looked 
for it elsewhere, depending on the nature of the difficulty. Even 
if useful additional information was not produced by this means, 
the participants usually elected to continue, using their best 
guess as to the subject or word meaning in question. 
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PIGURE 1: 
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Often, as soon as a question was asked, a possible or probable 
answering source came to mind, seemingly without any other mental 
processes being performed. This Is what Is represented In sub- 
routine B. If this Immediate response failed to produce the 
answer, the participants typically went on with other parts of 
the procedure. Likewise, sometimes Information In the question 
Itself Indicated a choice of current periodical Information, even 
before a subject classification had been made. As is shown at C, 
when this happened the participants followed the subroutine out- 
lined in Gl-4. 

It will be noted for each subroutine that if it failed to 
produce an answer, the participant needed to decide whether or 
not to continue searchings. For a "yes" decision, a return to 
A is indicated. While continuing a search after unsuccessfully 
executing a subroutine often involved asking the "patron" for more 
information, as well as reclassification by subject, in other 
cases these early decisions were not repeated, and the skip was to 
another part of the procedure. For a "no" decision to the 
question of continuing, a skip to X is shown. This indicates that 
at that point the participant would, in the real reference 
situation, have decided whether to refer the patron to another staff 
member, to take his name for contacting with the results of a 
later search, or to end the search completely. On the test 
questions the procedure ended here. In most instances of the step 
labeled "Answer found?" the participant would consult the "patron" 
as to the suitability of the answer and proceed accordingly, as 
is usually the case in the actual reference situation. 

The box marked D represents a key operation and one that was 
quite complex. The general or specific subject that was chosen 
to characterize the question and its eventual answer was usually 
the most important factor in making the subeequent decisions of 
the answering process. As well as being important to the 
initial attempt at finding an answer, reclassification by subject 
was often central to subsequent new approaches or attempts. 

In most instances a type of material, e.g., reference, cir- 
culating, magazines, was chosen before the necessity for con- 
sulting a catalog or index was considered. However, subroutine 
E is included to represent those several cases where participants 
decided first to check the card catalog under the chosen subject 
to see if the library had any books on the subject and if so, 
whether they were in the reference or circulating collection. 

Having made a subject classification and having failed to 
find a specific citation from the catalog, if the latter 
operation was performed, most participants on most questions 
next considered the reference collection as a potential place in 
which to find the answer (box F). The decision "yes" or "no" 
depended on many factors,, but mainly on two general considerations. 
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First, quite often the preliminary decisions pertaining to a 
question were made while the participant glanced over the ready 
reference collection or the general reference works such as 
encyclopedias. The probability that time might be saved by 
consulting one or more of the sources at hand was used in making 
decision F. The second consideration was a quick mental 
assessment of the contents of the appropriate subject section of 
the reference collection, in relation to the question. Depending 
on this assessment, the decision at F was "yes" or "no." Of 
course, in some instances, once a subject categorization was made, 
a specific reference book came to mind, and the decision at F was 
automatic. 

If the participant decided to consult reference sources and 
no general source or other specific source came to mind, then 
typically he went to the appropriate subject section of the 
reference shelves, with location help from the catalog if needed, 
to look for potential answering sources. Sometimes seeing a book 
on the shelves served as a reminder of it as a possibility. More 
often books were chosen for consultation on the basis of their 
titles or other characteristics. Sometimes none of the sources 
were judged worth consulting. 

The placement or sequence of decisions 6, H, 1 (and to some 
extent even F) on the chart was rather arbitrary, since these steps 
combined represent one of those complex decisions involving 
several alternatives, which were broken down into components for 
charting. The decision to pursue the answer to a question in 
either books, magazines, or pamphlets rested on many judgments 
concerning both the question and the materials. 

While the subroutine for periodicals (G) is fairly simple 
and straightforward, one common variation should be mentioned. 
Reference works which are essentially indexes or keys to periodical 
literature often contain considerable substantive Information 
themselves, in addition to the mere citations to articles. Thus, 
sometimes the answer to a question can be ascertained without 
going to the cited articles. This happened so seldom in this 
study that it was deemed unnecessary to expand subroutine G to 
account for it. 

Subroutine H is shown with broken lines to indicate that the 
decision to consult pamphlet sources was made at a number of 
places in the sequence of decisions, from quite early to the last 
resort, depending on the individual participants. Subroutine I 
is quite similar to F, outlined above. Here also, sometimes a 
specific circulating work came to mind upon classification by 
subject, in which case the participant skipped frcM D to 12. 

More often, the participant went to the appropriate shelves of 
the circulating collection and looked over the titles there. 

Based on a judgment as to the probability of their producing 
the answer, one or more books were consulted. 
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Using this general procedure, the 18 participants each attempted 
18 or more of the 29 test questions, for a total of 424 searches. 

In an attempt to gain more specific information on how the search 
process was used by the participants, each observation schedule 
and interview guide was examined, and each procedure step that had 
been observed or reported was coded, according to the step numbers 
on the flow chart, and recorded on data sheets. Due to the 
necessarily brief and informal nature of the interviews concerning 
the quest ion- answering process, it was difficult to be sure of the 
accuracy of the information on the unobservable mental processes 
and decisions. This brought into question the figures for the 
number of individual steps taken by participants, so that these 
figures are not reported. 

Since it rested more securely on observed behavior, the 
identification of subroutines resulting from the decisions is con- 
sidered more accurate and can be reported with more confidence. 

A surbroutine consisted of the steps that resulted from a "yes" 
decision at one of the basic decision points represented on the 
flow chart. For example, each time a participant made a "yes" 
decision at F, part of the steps numbered FI to Fll were executed, 
depending on subsequent decisions. Each such case was tabulated 
as a use of subroutine F. A "yes" decision at C and the resultant 
steps were coded as subroutine G, and operation D was included in 
whichever subroutine immediately followed it. 

Table 19 shows the number of times each subroutine was used. 

One pair of columns shows the number of times each subroutine 
was performed as the first or only phase of the search, indicating 
that the decisions at earlier points on the chart were "no" or 
were skipped. The second pair of columns shows the number of times 
each subroutine was performed as a second or later phase in a search. 
Since all pariticpants attempted to answer questions 1-18, figures 
on the searches for these questions use information from the 
largest number of searches that are comparable across all participants. 
Figures based on these 324 searches are reported in this table and 
tjjj^oughout the remainder of this chapter . Table 20 shows the number 
of searches that were composed of specific numbers of subroutines 
from one to eight. As can be seen a large number of questions 
required only one subroutine. 

While it is interesting to see the relative number of times 
each subroutine in the general reference procedure was used and the 
number of subroutines required to answer the test questions, no 
attempt should be made to generalize the relationship shown here 
to actual reference work. Although the test questions were made as 
typical as possible of the factual questions usually asked of medium- 
sized public libraries, it will be remembered from Chapter II that 
among the criteria used for choosing specific questions were the 
probability that the answers would require the use of one or another 



TABLE 19 



NUMBER OF TIMES EACH SUBROUTINE 
WAS PERFORMED 





As First Step 


As Subsequent Step 




Total 


Subroutine 


0. 1-18 


0. 1-29 


0. 1-18 


0. 1-29 C 


1. 1- 


18 0. 1-29 


A 


44 


58 


15 


17 


59 


75 


B 


50 


53 


- 


- 


50 


53 


E 


7 


8 


9 


10 


16 


18 


F 


161 


226 


88 


136 


249 


362 


G 


8 


14 


49 


57 


57 


71 


H 


5 


9 


8 


8 


13 


17 


I 


49 


56 


109 


131 


158 


187 


Total 


324 


424 


278 


359 


602 


783 


type of material and 
of the questions. 

NUMBER 


the expected general difficulty 
TABLE 20 

OF SEARCHES THAT WERE COMPOSED 


or complexity 
OF 



SPECIFIED NUMBERS OF SUBROUTINES 



Number of Subroutines 


Number of Times Required 
Questions 1-18 All Questions 


1 


176 


227 


2 


84 


114 


3 


32 


41 


4 


11 


19 


5 


14 


15 


6 


2 


3 


7 


4 


4 


8 


1 


1 



Total Subroutines Used 602 783 

Average Number Per Search 1.86 1.85 




Table 5 in Appendix A shows the number of subroutines required 
by each participant on each question. Data presented in Table 6 in 
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Appendix A shows the number of times each participant used each 
subroutine on questions 1-18, along with totals and percentages 
for each subroutine. Some relationships and patterns are 
apparent from these tables, e.^., trained participants tended 
to use subroutines for reference materials more frequently than 
untrained. These will be discussed later in this chapter when 
variations in answering procedures are analyzed. 



Variations in Reference Procedure 



While it is true that the answering procedures used by all 
the participants were similar in their general outlines, there 
were variations that can be related to differences in answering 
efficiency. In order to bring out these relationships, further 
abstraction was necessary. The procedure steps were classified 
into six types, across subroutines: 1) steps for gathering 

added information in order to aid in seeking answers, 2) steps 
involving recall of specific tools as potential answering sources, 
3) classification or categorization steps, 4) steps involving the 
use of catalogs and indexes to the library collection, 5) source 
selection steps, and 6) source use steps. The answering procedures 
used by each participant on each question, as represented on the 
observation schedules, were examined closely, and the 
differences in procedure that were judged crucial to differential 
efficiency were classified according to the six step types. 

Each type of answering step will be discussed in turn below, 
regarding the skills involved in the efficient execution of the 
step type, its importance to differential efficiency in this 
study, and participant characteristics related to its efficient 
execution. 

Information gathering steps . --The frequency with which 
reference questions, as first asked, contain too little infor- 
mation for reference staff members to formulate the execute 
searches for the answers is one of the problems most often 
mentioned in reference literature. Steps taken to add to the 
information contained in the question, in order to conduct a more 
efficient search for the answer, are those included in this 
classification. As represented on the flow chart, the steps of 
subroutine A were included here, as well as the accumulation 
of additional information as other phases of the search proceeded. 
The successful execution of this type of step was based on the 
ability to recognize that added information was necessary to, or 
might enhance, attempting a search, and on skill in procuring 
such information, w^hether from the patron or from materials in 
the library. 

Most of the questions in the test set were fairly straight- 
forward and clear. Thus, the necessity or opportunity for asking 
or seeking additional information to enhance the search process 
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did not arise often. In those cases where participants needed to 
seek additional information to answer a question efficiently 
they did so successfully. While the untr;ained participants 
needed to take such steps more often, they were taken quickly 
and did not cost appreciably in overall answering time. It 
should be emphasized again that this is only a reflection of 
the nature of the test questions used. To conclude from these 
results that the emphasis placed by many reference librarians and 
teachers on the importance of the so-called reference interview, 
or the interplay between librarian and patron, is wrong would 
be to exceed the data provided by this study. 

Specific tool response . --This type of step included those 
instances where a specific tool came to mind as a potential answer- 
ing source. Steps B3, F6, and 15 from the flow chart were 
classified here. The basis of succefis with these steps is 
difficult to specify, since they involve mainly mental processes 
almost at the subconscious level. For the participants of this 
study, success seemed to rest on familiarity with many information 
sources, as well as on the ability to abstract from questions 
significant features by which to call specific sources to mind 
in a sort of stimulus -response fashion. 

The causal element of the study hypothesis would predict 
that the untrained staff member would turn to remembered specific 
sources more often than the trained librarian; would do so 
earlier in the answering procedure, i.e., after fewer steps of 
classification and planning; and would have a lower rate of success 
with them. To see if this prediction held true, each time a 
participant reported thinking of a specific source prior to 
looking over the shelves or consulting a catalog or index on 
questions 1-18 it was tabulated as a specific tool response and 
as having been either successful or unsuccessful. The results 
are shown in Table 21. The number of responses reported on the 
table are divided between those immediate responses re- 
presented on the flow chart by subroutine B and the responses 
resulting from or following some classification steps. As can 
be seen, these data do not support the prediction. As a whole, 
the performance of the trained and untrained participants was 
quite similar in this area. There were some variations from 
participant to participant, especially in the net sources 
successfully consulted. The effect of these variations will be 
discussed below. 

Consulting specific information sources that came to mind 
took little time and thus seldom slowed the answering process 
appreciably if a consulted source failed to produce the answer. 

On the other hand, in many cases successful specific tool responses 
saved time and increased efficiency. These successful responses 
were fairly evenly divided between trained and untrained participants 
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and often involved information sources that were prepared by the 
local library staff. The participants who worked at the 
reference desk more hours per week seemed to enjoy some advantage 
in such specific tool responses, perhaps due to more frequent 
use of the tools. 



TABLE 21 

NUMBER OF SPECIFIC TOOL RESPONSES 
ON QUESTIONS 1-18 



Subroutine B^ After Classification Total Net 
Ijl I ^0t + Net Used Total 



Trained 

Participants 


23 


4 


19 


47 


30 


17 


104 


36 


Untrained 

Participants 


21 


4 


17 


46 


25 


21 


96 


38 


Totals 


44 


8 


36 


93 


55 


38 


200 


74 



*+ = Successful; - = Unsuccessful 



Classification steps: by sub1ec .t.--The procedure steps designated 

as classification steps were those involving decisions as to what 
subject areas the question and its eventual answering sources might 
fall into and what kinds of library materials (i.e., reference books, 
circulating books, magazines, or paiaphlets) were most likely to produce 
the answer quickly. For purposes of reporting findings, this group 
of steps was divided into two subgroups, those steps involving clas- 
sification by subject and those involving classification by type of 
material to be consulted. 

The classification of a question by subject was a very important 
and complex step in the search pjrocedures. In general, what seems 
to have been most important to success here was not only the ability 
to use the terms of the question, plus additional information if 
and when sought, to make an initial subject classification, but also 
the ability to use clues from the reference system (e.g., cross 
references in indexes or catalogs, or statements in consulted works 
that failed to provide a complete answer) to broaden, narrow, or change 
the subject as the search proceeded. For the 324 searches on 
questions 1-18, some 370 subject classifications or reclassifications 
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were reported or inferred from observations. Excluding some 40 
searches where the immediate response was to select a specific 
book prior to subject classification, this amounted to about 1 1/3 
subjectr tried per search. The trained participants as a group 
used a 1 » more subjects than did the untrained. 

Classification by subject was among the most important step 
types in accounting for differential efficiency. Around 20 
percent of the crucial procedure differences were classified here. 

The untrained participants had considerably more trouble with 
subject classification than did the trained. The main difficulty 
for the untrained staff members was the use of subjects that were 
too broad or general, e.g., "biology" for question number 3 
(life span of the Red Squirrel), rather than the more narrow and 
useful "wild animals" or "mammals." This suggests that such 
matters in library school courses as the emphasis on close 
subject classification and the principle of specific subject 
headings gave the trained participants an advantage in this area. 

Particularly successful subject classification of questions, 
contributing to high efficiency, seemed to be closely related to 
what might be called familiarity with the library collection. 

Close familiarity with the collection seemed to provide both a 
wider set of alternative subjects and the knowledge needed to 
choose quickly and correctly among them. This was indicated by 
the fact that in the majority of instances, where subject 
classification difficulty was identified, the participants used 
one classification too long, perhaps for the lack of alternative 
subjects, gave up on questions without trying alternate subjects, 
or consciously rejected a potential subject because of a 
mis judgment of the contents of the library's collection in that 
subject. 

Classification steps; by type of material . --Decisions as to 
what type of material to consult were also very important to 
efficiency in answering the test questions. From the discussion 
of subject classification above, it can be noted that the 
relation between classification by subject and classification 
by type of material was not related in as simple and sequential 
a manner as the flow chart would indicate. Often these two 
processes were performed in one complex operation, with subject 
classification utilizing assessments of appropriate sections of 
various types of materials and the choice of a material type 
resting on the subject finally selected for priority. Success 
in choosing a type of material seemed to rest on the ability 
quickly and accurately to judge the contents of the various 
collections of types of material in relation to such character- 
istics of the question as subject, recency of information needed, 
approximate level of detail or specialization called for, etc. 
Within the general categories, more specialized type classifications 



were often made (e.g., field guide or handbook) necessitating an 
ability to judge the contents of these also. 

Classification by type of material was also very important 
to differential efficiency, again accoutiug for around 20 percent 
of the crucial procedure differences. Trained participants had 
fewer differences that were judged to have caused low efficiency and 
more that contributed to high efficiency than did untrained. In 
most cases of difficulty due to classification by type of material 
the untrained participant rejected reference materials on the basis 
of a misjudgment of the contents of the reference collection in the 
subject field involved. For example, on the question concerning 
the Red Squirrel an untrained participant might have said, "I 
don't believe any of our reference books on animals would contain 
such information," while a trained participant would say, "I 
believe a field guide on animals in the reference collection would 
contain such information." Untrained participants also tended to 
continue searching unsuccessfully in one type of material longer 
than did the trained staff members, to the detriment of efficiency, 
perhaps from a lack of alterantives thought of or judged feasible. 

Particularly efficient type-of-material categorizations were 
often credited by the participants to regularly performed process- 
ing duties with the type of material involved. Along this line, 
it is interesting to note that eleven of the thirteen uses made 
of pamphlets on questions 1-18 were by participants who were 
or had been responsible for processing pamphlet materials. 

Many cases of differences in classification by type of material 
were judged to be due to peculiarities of the libraries involved. 

For example, in one library a substantial part of the circulating 
collection was shelved on the floor above that containing the 
reference desk and collection. The participants in this library 
chose to consult circulating materials at the shelves in these 
subject areas less often than did those at other libraries, some- 
times hampering efficiency. 

Catalog or index use steps . — The ability to use the bibliographic 
and indexing apparatus of a library collection is considered to be 
among the most important skills of the reference librarian by most 
reference practitioners and teachers. Due to the relative simplicity 
of the test questions and the heavy reliance by all participants 
on memory of the shelf locations for various subjects and choice 
from among the tools found on the shelves, catalog and index use 
assumed somewhat less importance in this study than might have been 
expected. Success in such use for the study participants depended 
on the ability to translate the terms of the question or the subject 
classification thought of into the formal language of the catalog 
or index consulted. Also important were the readiness and ability 
to use cross references and to use the various pieces of information 
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on B. cntnlog card or in a citation to decide quickly and accurately 
if the cited source might answer the question. 

The rationale on which the causal element of the study hypothesis 
was based would suggest that the trained participants would make 
greater use of the bibliographic equipment of the library in order to 
identify sources for answers to the questions. Data from this study 
did not support this expectation. The 18 participants consulted the 
card catalog 103 times on the first 18 questions. Of these 103 
uses of the catalog, 43 were by trained and 60 by untrained participants. 
Within each group, the uses were fairly evenly spread, except that 
participants in the library mentioned in the section above used the 
catalog more often, to compensate for the inconvenience of using shelf 
location for those materials shelved at a distance. Periodical index 
use constituted almost all of the index use, of which there were 26 
instances by trained participants and 35 by untrained. This latter 
might be expected, since untrained staff members tended to use 
periodicals more than trained. 

Differences in use of the card catalog and indexes were judged 
to have caused differential efficiency in about 10 percent of the 
cases. Such differences in procedure were fairly evenly divided 
between trained and untrained participants. Use of the card catalog 
caused the untrained participants the most trouble. They had 
difficulty translating the terms of the questions into the headings 
of the catalog and in using cross references. The trained participants* 
troubles were more in the area of index use and took the form of 
difficulty in arriving at the appropriate index terms to use and lack 
of thoroughness. 

Source selection . --In those cases where immediate responses or 
classification decisions did not bring to mind specific sources to be 
consulted, potential answering sources had to be chosen from among 
those on the reference or circulating shelves examined. Efficiency 
in this type of step seemed to rest on the ability to distinguish 
between similar sources on the basis of outward characteristics, such 
as title, or to remember past usefulness of a source upon seeing 
it on the shelf. 

Included in this category of steps was the decision to consult 
a general source or sources, such as an encyclopedia or almanac, 
since this decision was often or usually made while glancing over 
the general or ready reference section of the reference collection. 

Thus, to an extent, the decision to consult such sources was a matter 
of selecting one source fro«i several seen. Since the fact that 
these decisions had been made was largely inferred from observed 
behavior, and since the sources involved were quite familiar to the 
participants, classification was rather arbitrary, and some of the 
decisions to consult general sources might have appropriately been 
included among specific tool responses or type-of-material clas- 
sifications. . At least, a close relationship exists among these three 
types of steps. 
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For the first 18 questions the participants decided to consult 
general sources 55 different times. Of these 55 decisions, 26 were 
by trained and 29 by untraineu participants. For both groups of 
staff members nine such decisions produced correct answers, and the 
remainder failed to do so. The decision to consult a general 
source was ostensibly taken in an attempt to same time. That is, 
a participant would often say something like, "While we are here-, 
let*s check the almanacs." However, the investigator often felt 
that the general work was being consulted while the participant 
attempted to think of the best subject and type of material 
approach to use, a device suggested in some reference texts. Thus, 
the choice of a general reference tool was perhaps based on^a 
combination of a judgment of the probability of that source s 
containing the answer and the immediate perception of other possible 
approaches. Consulting general sources seemed to work somewhat 
more to the advantage of the trained participants, perhaps as a 
result of a more accurate assessment of the contents of sources, 
to be mentioned again below, and as a restilt of more facility^ 
in subject and type-of-material classification, as mentioned in 
previous sections. 

The remaining steps classified as source selection included those 
steps where a source (other than a general source) not previously 
thought of was selected from a shelf for consultation. This excludes 
those instances classified as specific tool responses. Out of 
324 searches by the 18 participants on questions 1-18, 171 searches 
involved such selection. For these, some 364 separate sources were 
chosen and consulted, 176 by trained and 188 by untrained participants. 
Of these, 53 (43 percent) of the choices by trained participants 
produced correct answers, as opposed to 59 (46 percent) for the 
untrained. 

Source selection differences were judged to have contributed to 
between 15 and 20 percent of the cases of differential efficiency. 

Such cases were evenly divided between trained and untrained parti- 
cipants and seemed to lack any other pattern that could be related 
to independent variables. Two participants who showed considerable 
difference in their choice of answering sources, sometimes to the 
detriment of efficiency, worked in a library where space shortage 
had forced the removal of some materials from the regular shelves 
for shelving in less accessible areas. 

Source use. --Having remembered or chosen a tool as a potential 
source for the answer to a question, the ability to find the answer 
in it or /to decide that the answer was not there was important to 
the efficiency of the study participants. Success at this point 
seems to have been based on the readiness to use a .d skill in 
exploiting the explanatory and other auxiliary sections in unfamiliar 
tools, the use of techniques to increase speed in getting at the 
information needed, and thoroughness. 
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The efficiency with which chosen sources were used was judged 
to have been the contributing factor in some 20 percent of the 
instances of differential efficiency. The untrained participants 
had much more difficulty in source use than did the trained. 

Three major types of use difficulty were noted among the 
instances of low efficiency between members of pairs due to 
source use. The chief of these was the failure to use the table 
of contents, "how to use," and other auxiliary sections of an 
unfamiliar or complicated work to aid in it use. Often both 
members of a pair selected the same source and both were unfamiliar 
with the arrangement of sections and placement of information needed, 
e.g., the roanuf acturers ' and advertisers' directories used to 
answer question number eleven. While the more successful member 
of the pair used the auxiliary material to help locate the needed 
information quickly, the other participant attempted the more direct 
approach of leafing through the volume, failed to find the needed 
information quickly, and sometimes incorrectly decided it was not 
there . 

A second difficulty involved failure to use techniques to 
increase the speed of the use of information sources. Some of these 
are in the nature of "tricks of the trade." For example, a help- 
ful device used by several participants was always to start with the 
last bound volume when using a cumulated reference set for recent 
information, before taking the* time to consult each smaller unbound 
issue. Failure to use this technique cost some untrained parti- 
cipants time, as did failure to use the general indexes to 
encyclopedias in some cases. 

Finally, there was the lack of thoroughness, e.g., the use of 
one index entry when taking the time to see and use another farther 
down the page would have been more helpful, or the use of a table 
without reading the heading or explanatory text carefully. 

Two further points need to be made. The two trained participants 
who had their training from unaccredited school library program 
showed source use patterns more similar to the untrained participants 
than to the other trained participants. These staff members reported 
having studied a fairly restricted range of types of materials in 
their reference courses and little or no emphasis on techniques for 
their use in reference work. Even though some of the tools which 
the other trained participants used more successfully were reported 
not to have been studied in library school, there seems to have been 
some generalized effect on source use from studying a broad range of 
types of information sources. 

Also, it will be remembered from Chapter II (pages 36 and 37) 
that larger differences in efficiency between members of pairs tend- 
ed to be associated with smaller library collection sizes. Analysis 
of differences in answering procedure across pairs indicated that 
much of the crucial difference was in the area ol’ source use. In the 
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larger collection, when a participant made a subject classification 
of a question, the shelf cons ' ' “ ' ' i or several useful 



collections greater use had to be made of more general sources, 
demanding more skill in selecting appropriate general tools, perhaps 
based on more thorough knowledge of their contents, and greater 
abi?:ity to get at the information in them through indexes, etc. 
Crucial differences in procedure due to source use were more 
prevalent among participants in smaller libraries. 



This chapter represents an attempt to present information on 
the nature of the re. tionships betwen the independent variables 
and reference performance by examining the answering procedure used 
by the participants. Contrary to the prediction of the causal 
factor of the hypothesis, the trained and untrained participants 
were found to have used the same general type of procedure. This 
procedure was composed of a series of several basic and complex 
decisions, out of which grew various operations or subroutines, 
depending on the decisions reached. The difference between the more 
efficienct and less efficient participants, then, was not to be 
found in the general search procedure used, but rather in the 
usefulness of the decisions reached at various points in the procedure 
and the skill in performing the operations called for by the de- 
cisions. 

In order to analyze differences in procedure seriously affecting 
performance efficiency, the many steps of the general search procedure 
were classified into eight basic step types and sub-types including 
both decisions and operations: 1) Information gathering steps, 2) 

specific tool responses, 3) classification by subject, 4) classification 
by type of material, 5) catalog or index use, 6) source selection 
from among general reference tools, 7) source selection at the shelves, 
and 8) source use. 

Classification by subject, classification by type of material, 
and source use were found to be the types most frequently associated 
with differences in search procedures that resulted in differential 
efficiency. For each of these step types, variation in successful 
execution was most closely related to variations in professional 
education, among the independent variables discussed in Chapter 11. 

That is, professionally trained members of pairs had fewer crucial 
search differences associated with high efficiency than did untrained 
members of the sane pairs. Indications of relationships between 
other independent variables and efficiency were found. Time spent 
per week actually answering reference questions seemed to increase 
success in remembering specific tools as answering sources, particularly 
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those prepared by the library staff. Non-question- answering duties 
regularly performed were noted to be related to success on clas- 
sification steps, particularly classification by type of material 
to be consulted. 

Other independent variables were related to procedure, and 
thus to efficiency. - Variations in the libraries in which the 
participants worked were related to crucial differences in several 
step types. There were indications that small collection size in- 
creased opportunity for the close familiarity with the collection 
that enhanced classification by subject and type of material. In 
the two libraries with some materials shelved at a distance from 
the reference desk or shelved out of normal sequence, type-of- 
material classification and source selection was affected. The 
size of the collection seemed to make differential demands on skill 
in source selection and use, thereby affecting differences in 
efficiency between trained and untrained members of pairs. 

Conclusions that can be drawn from these findings, as well as 
the practical implications for reference service, will be 
discussed in the following chapter. 



CHAPTER IV 



SUMMARY AND CONCLUSIONS 



The purpose of this study has been to gather and examine data 
on the relationship between formal library education and effective- 
ness in answering reference questions, as well as to gain some 
empirical information on the operation of other variables in the 
reference situation. The method employed to do this was a series 
of field studies in which nine pairs of reference staff members 
in medium-sized public libraries were asked a set of test 
reference questions and observed carefully while they attempted 
to find answers to them. The members of each pair were as similar 
as possible with regard to such variables as amount of under- 
graduate education and years of reference experience, while 
differing in amount of formal library training. 

The data collected was used to test not only the study 
hypothesis that the trained librarians would be able to answer 
significantly more questions and would do so more rapidly, but also 
some alternative hypotheses that might have explained the differences 
in performance found. In addition, the effect of other variables 
that were found to be operative in the reference situation were 
discussed. Finally, in an effort to specify in more detail the 
nature of the relationships found to exist, variations in the 
procedures used by the study participants to find the answers to 
the test questions were examined. The balance of this chapter will 
summarize the findings of the study, specify some of its limita- 
tions and assumptions, outline Tome conclusions that can be drawn 
from the findings, discuss some practical implications of the find- 
ings and conclusions, and suggest some directions for further 
research. 



Summary of the Findings 

1. The ratio of questions answered correctly to all those 
attempted, used as an indication of the participants* accuracy, was 
not significantly different between trained and untrained parti- 
cipants. 

2. Trained participants took significantly less time per question 
attempted than did untrained participants. 
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3. When speed and accuracy were combined to yield a figure 
for the time required per correct answer for each participant, used 
as an indication of performance efficiency, the trained participants 
were found to have performed significantly more efficiently than 
the untrained. Differences ranged from 1.02 minutes per correct 
answer in favor of the untrained member of a pair to 3.24 minutes 
per correct answer in favor of the trained member of another pair, 
with a mean difference of 1.23 minutes in favor of the trained 
participants. 

4. When the age of the participants was used as a alternative 
independent variable to relate to the differences in efficiency 
between members of pairs, the results were not statistically signi- 
ficant, thus weakening the credibility of one alternatice hypothesis. 

5. The number of years that had elapsed since the end of the 
participants' formal education was also used as the independent 
variable in an alternative hypothesis. No significant results to 
challenge the study hypothesis were found. 

6. When the relative amount of responsibility for the selection 
and handling of reference books was related to efficiency, a signi- 
ficant relationship was found. While this alternative hypothesis 
could not be destroyed, it did not account for the differences in 
enough pairs to supplant the study hypothesis. 

7. The number of hours per week spent answering reference 
questions was also advanced as an alternative independent variable. 

Some apparent positive relationship between hours assigned to the 
reference desk and efficiency was noted. The fact that some crucial 
differences in answering procedure were associated with this 
variable strengthened the credibility of this alternative hypothesis, 
though again not enough of the difference was explained thereby to 
justify rejecting the study hypothesis. 

8. Attention was also given to the relationship between several 
variables and the size of the advantage in efficiency had by the 
trained members of the pairs. The variable most strongly related to 
the size of difference in efficiency was the amount of difference 

in responsibility for selecting and handling new reference books. 

The wider the difference in such duties between the members of the 
pairs (in the direction of the trained members' having more duties), 
the larger was the trained members' advantage in efficiency over 
their counterparts. The rank order correlation coefficient between 
these two variables was .767. When differences in answering procedure 
related to differential efficiency were studied, these duties, as well 
as other regularly performed duties connected with information materials, 
were found to be associated with success in specific tool responses, 
classification by subject, and classification by type of material to 
be consulted. 

9. A rank order correlation coefficient of -.642 was found 
between the amount of in-service training and direct professional 
supervision reported by each untrained participant and the size of the 
difference in efficiency between that participant and the trained 
counterpart. 
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10. A decrease in the size of the adult collections with which 
the pairs worked was correlated with an increase in the amount of 
advantage in efficiency the trained participants had over the 
untrained. The rank order correlation coefficient here was -.533. 
Findings from the study of the answering procedures used by the 
participants indicated that this relationship was due to increased 
demands made by the smaller collections on the source use skills 

of the participants. Other differences from library to library 
were found to affect efficiency, particularly by affecting type-of- 
material classification and answering source selection. 

11. Not enough difference in the formal reference education 
the trained participants had received existed to relate this variable 
to efficiency meaningfully. The two participants with degrees from 
unaccredited programs were at some apparent disadvantage in efficiency, 
and this was confirmed by the study of reference procedure. However, 
the differences were not clear-cut enough to judge their significance. 

12. The basic or general procedure used in searching for answers 
to the test questions was found not to b^essentially different for 
trained and untrained participants. For both it consisted of a 
series of more or less ad hoc decisions, as opposed to planned 
strategy, and operations resulting from the decisions. Differences 

in answering efficiency were due to differences in the appropriateness 
of the decisions reached and the skill with which the operations 
decided upon were executed. 

13. Of the 191 books or sources, excluding magazines, in which 
the trained participants found answers to all the questions attempted, 
28 percent were published after the participants using the sources 
finished library school. Another 25 percent would not have been 
studied in reference courses because they were circulating materials 
of types not considered in reference and bibliography courses, at 
least not by title. Finally, for many of the remaining 47 percent the 
participants reported not having studied them in library school 
courses, or could not remember whether or not they had studied them. 

14. Differences in classification by subject, classification 
by type of material to be consulted, and source use were the most 
important types of procedure differences associated with differential 
efficiency. Each of these accounted for some 20 percent of the 
total differences in procedure judged crucial to efficiency. 



Limitations and Assumptions 

Any conclusions or generalizations based on the findings sum- 
marized above should be drawn only after the limitations and assumptions 
implicit in the design and method of this study have been made 
explicit. Those that are most important are outlined below. 

1. The libraries in which the participants worked were all public 
libraries, and the test questions used were drawn from public 
libraries. Findings would be applicable to other types of libraries 
only to the extent that their reference situation are similar to 
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those of public libraries, a point on which little objective information 
is available. 

2. The reference situations studied were all in medium-sized 
central libraries. No very small or very large libraries were 
studied, nor were any branch library operations Included. 

3. Only general reference departments were studied. The 
applicability of the findings to subject departmentalized reference 
systems is, therefore, unknown. 

4. Only a small number of cases were studied, and these were 
not randomly selected or claimed to be systematically representative 
of any wider population. There is no reason, however, to believe 
that the libraries or participants studied were atypical of medium- 
sized public libraries. 

5. The test questions asked for factual, not interpretative or 
judgmental, information. While this type of question has been found 
to comprise, the major part of the day-to-day work of a reference 
department, findings based on the test questions do not allow 
generalization to all types of reference questions. 

6. Any generalization from the nine participants with fifth-year 
library degrees to support a hypothesis concerning "trained” librarians 
rests on the assumption that reference education has been similar for 
all or most trained librarians, including all those studied. As 
pointed out in Chapter II and on Table 15, page 35, the reference 
training received by the participants was fairly similar. In addition, 
the participants represented a fair cross section of library schools, 
having attended two different unaccredited schools and five different 
accredited schools. The literature of reference instruction also 
indicated a considerable similarity among the training agencies.^ 



Conclusions 



Within the limitations above, some conclusions can be drawn. 

1. The study hypothesis, which predicts that professionally 
trained librarians will be able to answer a larger proportion of 
information requests and will do so in less time than will untrained 
staff members, is supported by the data from this study. The 
predicted differences were found to exist among the participants, 
and reasonable alternative explanations for the differences were 
found not to be as tenable as the study hypothesis. 

2. The contention by some that professionally trained staff 
members will be more thorough and cognizant of the currency and 
accuracy of the factual information they give in answer to questions 
finds little or no support in this study. On the questions where 
the most opportunity for this distinction arose, both trained and 
untrained participants showed a tendency to accept the first answer 
found in print, and the difference in their accuracy was not 
significant. 

3. The significant differences were on the variable of 
answering speed. Even here the differences were not great. However, 
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a reference librarian* s day is made up of many questions, each 
occupying a small segment of time. A small difference of each 
question can add up to a large difference over the course of a day. 

Even one added minute per question can be a considerable time to a 
patron who must wait through several other questions before his own 
is answered. 

4. To the above limited extent, then, the reference staffing 

principle, noted in Chapter 1, that only trained librarians should 
be assigned to answering reference questions, is supported by the 
findings of this study. ^ 

5. On the other hand, findings of the study also show that 
staff members lacking formal library education can answer 
accurately and quickly a wide range of factual refernece questions 
and can, under some circumstances, perform as efficiently as 
professionally trained librarians, or even more so. 

6. The factors that were most strongly associated with the 
smallest differences in favor of trained members of pairs, or with 
differences in favor of the untrained members, all involved 
mechanisms that allowed increased familiarity with the information 
contents of the library collections — in-service training, respon- 
sibilities for the selection and handling of reference books, and 
other duties having to do with materials used in reference work. 

These factors also are indicative of close integration of untrained 
staff members into the day-to-day operations of the reference depart- 
ments, over and above merely manning the desk and answering 
questions. 

7. The knowledge on which increased efficiency in answering 
questions was based might be generally characterized as familiarity 
with the library collection. However, only a small proportion of 
the instances of higher efficiency were based on direct memory access 
to specific sources or titles and their contents. Rather, the 
important familiarity was in terms of knowledge of the subject 
classification mechanisms of the library, the relationships between 
the contents of various types of materials, the formal language of 

the catalog and indexes to the collection, in addition to a generalized 
skill in the use of information sources. 



Practical Implications 

The findings and conclusions of this study have implications 
to library administration and education, some of which are outlined 
below. 



1. As one partial answer to the shortage of professionally 
trained librarians, reference administrators should consider 
mechanisms for assigning untrained staff members responsibilities 
for answering at least factual reference questions. Such con- 
siderations would have to take into account the potential increased 
cost in time per question indicated by the findings of this study. 
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as well as other factors on which there is a lack of objective 
information. 

2. The opportunities for learning about reference materials 
and reference skills, both from formal in-service training and 
from the supervised performance of relevant duties should be made 
amply available to untrained staff members responsible for 
answering reference questions. 

3. A balance between amount of time assigned to answering 
reference questions and amount of time assigned to other regular 
reference-related duties should be maintained. Both seem to make 
important contributions to answering efficiency. Also, the fact 
that untrained staff members who were more closely involved in a 
variety of the day-to-day operations of the reference departments 
performed more efficiently might illustrate the standard adminis- 
trative principle that the staff member who feels that he is a 
contributing and vital member of a team is likely to be more 
productive.^ 

4. Since a large portion of reference work is done under the 
pressure of time, or under "rush hour" conditions, the principles 
of "least effort" and "least time," found to be very operative on 
the test questions, are undoubtedly a significant element in 
practical reference work. For this reason, it is important that 
time and effort saving devices, such as the ready reference 
collection and special arrangements of the reference collection, 
be designed to contribute to accuracy and authority of answers, 

as well as to the speed with they may be found. Otherwise, the desire 
to save time for the patron can come into conflict with the pursuit 
of the best answer to his question. For example, some libraries 
among those studied kept the standard general almanacs and 
Statistical Abstract at the reference desk near the telephone, 
while others omitted the latter. Participants at the libraries 
with only the almanacs at the desk used them more often and 
Statistical Abstract less often, sometimes to the detriment of the 
completeness or currency of the answers given. 

5. The fact that shelf classification and shelf location were 
relied upon heavily in reference work is an argument for the 
consistency and integrity of the classification of books in a 
library. This means also that definite policies with regard to 
what types of material are put into the reference collection and 
what types into the circulating collection should be made and 
followed. When major changes in classification designations for 
subjects need to be made, the reference staff should be made 
cognizant to them. 

6. Other decisions regarding the physical arrangement of 
libraries and library materials should take into account possible 
effects on reference efficiency. This is not to say that such 
considerations should outweigh considerations of patron convenience 
or reading encouragement by easy access, but the probable costs as 
well as potential advantages should be used in making such decisions. 
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7. Implications of this study for library education are 
particularly hard to specify. If it is the case that skill in 
answering factual reference questions can be adequately learned on 
the job, perhaps library school reference courses should deemphasize 
teaching these skills and concentrate on preparing librarians to 
administer reference departments for efficient service, to train 
and supervise reference assistants, and to answer more difficult 
types of questions. 

8. The importance in this study of the appropriate cat- 
egorization of questions with regard to subject and type of material, 
and the fact that direct memory access to tools studied in library 
school was much less important to differential efficiency, would 
indicate that even courses emphasizing skill in answering questions 
should teach more than detailed knowledge of the contents and use 

of specific reference tools. What the trained participants seem to 
have brought from library school, and what some of the more efficient 
untrained participants had been able to learn on the job, was an 
understanding of relationships and operations within the information 
contents of the library--relationships between general and specific 
subject classifications and subject terms and operations for 
moving among them, relationships between the types of information 
contained in different types of material and operations for getting 
to the information needed, relationships between various auxiliary 
and substantive sections of complex reference tools and operations 
for coordinating them. 

This is not to say that working knowledge of many basic and 
specialized sources was useless to the participanss of this study. 
Perhaps the practice of reference is somewhat analogous to the use 
of language. The knowledge of a large number of reference sources 
would compare to the possession of a large vocabulary; each is 
necessary to skill. However, articulate use of a language requires 
the effective command, whether conscious or not, of the rules of 
gr amm ar and skill in their application. Likewise, it might be 
argued, effective practice of reference librarianship requires 
knowledge of what has been called the "principles and operating 
characteristics of the bibliographic system."^ The process of 
finding the correct answer to a reference question has been charact- 
erized as traversing the world of published writing to the point 
where the desired information is presented in the form and style 
needed by the patron. 5 This study indicates that general knowledge 
of the world to be traversed is at least as important as specific 
knowledge of the points to be reached. 



Suggestions for Further Research 

More information on reference personnel should be pursued on 
a number of fronts, desirably in a coordinated fashion. Below are 
some ideas for replicating the present study and extending similar 
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research Into other types of reference situations, as well as some 
suggestions for research into other related areas. 

The present study should be considered exploratory or 
preliminary. Before its findings and conclusions can be accepted 
with real confidence, other studies along similar lines should be 
done to validate, supplement, or contradict it. Some needed 
refinements and improvements are evident. Most obvious is the 
need for more adequate control of the several variables studied. 

For example, conclusions as to the relationships between performance 
and such variables as in-service training, non- quest ion- answering 
duties, age, experience, and size of collection had to be very 
tentative because the proposed independent variables themselves were 
related to each other, and their individual effects could not be 
separated. 

Perhaps the eariest way to improve control on some of the 
variables would be to get information concerning them in advance 
of selecting the participants, so that differences on these 
variables can be taken into account in matching and choosing 
participants. Attempting to match on too many variables would 
probably not be feasible, but if, for example, information on in- 
service training is known in advance, the investigator can avoid 
having a group of cases where amount of in-service training varies 
with amount of experience > In any case, attention to more variables 
prior to selection would probably necessitate surveying a larger 
group of libraries for potential participants than those found in the 
the four- state area used in this study. Likewise, using information 
from this study concerning factors of library arrangement and set-up 
that affected reference efficiency, an investigator could construct 
indexes for scoring libraries and attempt to relate more precisely 
to performance other interlibrary differences besides collection 
size. 

Using the present study and considerably more preliminary study 
as a basis, research of more sophistocated design should be conducted 
so that variables could be controlled statistically, through such 
techniques as partial correlation. Random sampling should be used 
to choose participants. More precise measurement of the variables 
than rankings should be developed, since at least an interval scale 
level of measurement is required for such statistical manipulations. 
This would necessitate more rigorous definitions of some of the 
variables, as well as more attention to the reliability and validity 
of the measures. 

Research on the differential ability of trained and untrained 
staff members to answer reference questions should be extended into 
different areas from those considered in this study. One might ask 
if the same or similar relationships hold for other types of questions. 
Other test sets of questions could be constructed, using questions 
that are more difficult to find answers for and questions asking for 
interpretative or judgmental information, wherein the librarian has 
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to help the patron find and evaluate information on broader more 
open-ended questions. If it is found that untrained staff members } 
given appropriate on-the^job training and experience, can answer 
certain kinds of questions but not others, the next logical question 
is whether or not they can recognize the difference soon enough to 
avoid wasting time by attempting inappropriate questions. Some 
administrators hesitate to use untrained personnel to answer 
questions not because they feel that there are no questions that 
such persons can answer, but because of a fear that untrained 
staff members will not have sufficient judgment to know when 
they are out of their depth or when to refer a question to another 
staff member. The validity of this fear needs to be tested. 

Other sizes of libraries need to be studied, particularly 
very large ones. The same kind of study could be conducted in 
very large general reference departments with a great many 
specialized reference tools and with circulating collections spread 
over several, floors and probably in several subject departments. 

Results would quite possible be considerably different from those 
of the present study. Other types of libraries should also be 
studied. Reference staffing problems are no less severe in academic 
libraries than in public libraries, and yet academic libraries 
might be able to tap a ready supply of untrained reference assistants 
in college-educated student and faculty wives, as well as advanced 
students. Research studies need to be designed and executed, comparing 
the ability of subject specialists lacking library education and 
trained librarians with little subject background to do reference 
work in subject areas. 

In addition to replicating the present study, with improvements, 
and extending it to different reference situations, other areas 
are much in need of attention before the findings of studies like 
this one can be put to the best use. If it is true that at least 
some staff members lacking formal library education can answer 
reference questions quickly and accurately, how can this fact be 
put to the best use of the library? Management research needs to 
be done on procedures and techniques for screening questions and 
directing them to appropriately trained staff members or for smooth 
referral of questions among staff members. This is a difficult 
problem, involving patron comfort and public relations, which is one 
of the factors underlying the staffing principle examined in this 
study. The lack of a solution for it will stand in the way of the 
applications of findings from studies such as the present one. 

Also in need of attention are the in-service training and on- 
the-job experiences that can enable the untrained staff member to 
perform efficiently on reference questions. Only the most meager of 
information on this problem could be offered in this study. Success- 
ful programs in operation should be examined and described, and 
feasible ideas experimented with. Also, features of local reference 
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system arrangement and procedures that are most closely related 
to performance efficiency should be identified. Finally, the back- 
ground and personality characteristics of untrained reference staff 
members that are related to reference performance should be 
identified. Particularly interesting might be the attitudes of 
the staff members toward finding answers to questions. For example, 
staff members who emphasize reader advisory work and the pleasures 
of reading might well consider much emphasis on efficiency in- 
appropriate to books and their use. 

The implications drawn above with regard to formal library 
education for reference work are very much open to question and 
research. Even if the information and concepts to be conveyed are 
of a more general nature than details about specific sources and 
their uses, the best method for doing this might still be the 
rather inductive one of using a great many examples of specific 
informatics sources, from which the characteristics of the bib- 
liographic system can be generalized. Carsfully controlled 
experimentation with different methods of education for reference 
work should be carried out, as well as the examination and 
application of relevant research information from other disciplines. 
Research information about the knowledge and skills necessary to the 
practice of librarianship cannot be made useful until the means of 
conveying them to librarians are discovered. 

Finally and perhaps most important, there is need for a program 
of research into the operating characteristics of the system which 
has been evolved by society to communicate and, store the information 
that is always the object of reference questions. Even the limited 
findings of the present study indicate confirmation of the following 
statement by Freides: 



The really important knowledge demanded of the reference 
librarian is knowledge of relationships and patterns 
within a system: the characteristics of the universe 

of publication, the way in which the tools of our trade-- 
the handbooks, bibliographies, and so forth--reflect 
this universe, and the quantity and quality of distortion 
produced by the reflection; the way in which the contents 
of the writings are encapsulated and codified in indexes 
and catalogs, and the amount and kind of seepage taking 
place thereby; the relationship between basic research and 
summary, between scholarship and popularization, and the 
meaning of all these in terms of the reference inquiry. 



However, as Freides goes on to say, "many of the components of this 
. system are as yet unknown and many more are unarticulated." Careful 
continuing research in this area might well be requisite to 
understanding reference librarianship and staffing for its effective 
service. 
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Footnotes to Chapter IV 



^Samuel Rothesteln, "The Measurement and Evaluation of Reference 
Service," Library Trends , XII (January, 1964), 456-72. 

2 

Louis Shores, "We Who Teach Reference," Journal of Education for 
Librarianship , V (Spring, 1965), 238-47. 

3 

Joseph Wheeler and Herbert Goldhor, Practical Administration of 
Public Libraries (New York: Harper and Row, 1962), p. 235. 



4 

Thelma Freides, "Will the Real Reference Problem Please Stand 
Up?" Library Journal , XCI (April 15, 1966), 2208-12. 

^Ibid.; see also Mary N. Barton and Ellen F. Watson, General 
Reference Staff Manual (Baltimore: Enoch Pratt Free Library, 1950). 



6 

Freides, loc . cit . 
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lABLE 1 

SUCCESS ON EACH QUESTION BY PARTICIPANTS 
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TABLE 1 (Continued) 
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mSLE 2 



MINUTES REQUIRED ON EACH QUESTION 
BY PARTICIPANTS 
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lABLE 2 ( Continued ) 
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lABLE 3 

SUMMARY DATA ON QUESTIONS ANSWERED AND TIME TAKEN 
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kl 


13 


90.0 


18 


14 


B1 


15 


84.3 


20 


17 


B2 


15 


92.7 


18 


15 


Cl 


16 


45.8 


28 


27 


C2 


16 


66.2 


26 


23 


D1 


15 


90.2 


19 


17 


D2 


16 


77.6 


21 


20 


El 


15 


72.7 


25 


22 


E2 


14 


114.5 


16 


13 


FI 


15 


72.7 


25 


22 


F2 


17 


80.2 


25 


24 


G1 


16 


68.6 


27 


26 


62 


16 


89.0 


19 


18 


HI 


17 


55.2 


26 


25 


H2 


16 


101.0 


17 


16 


11 


14 


81.2 


19 


16 


12 


11 


91.2 


19 


13 



81 



TkBlE 4 

NUMBER OF ANSWERING SOURCES USED PER 
QUESTION BY PARTICIPANTS 



Question 

1 

2 

3 

4 

5 

6 
7 
3 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 
29 



Participant 

A1 A2 B1 B2 Cl C2 D1 D2 El E2 



2 2 
1 1 
6 6 
1 3 

1 4 

3 1 

1 3 

1 1 
1 1 

4 5 

1 3 
1 2 
1 1 

5 2 

3 1 

5 3 

2 1 

2 2 
3 5 

1 



1 2 
2 1 

5 3 
1 2 
2 2 
1 2 

1 7 
1 1 
1 1 

6 4 
1 1 

2 1 

5 2 

2 1 

5 2 

6 3 

5 2 

1 2 
3 1 

1 2 
9 5 

3 - 



2 2 
3 6 

2 5 

1 3 

1 1 

3 1 

1 5 
1 1 
1 1 

2 1 

1 2 
2 3 

1 6 
1 1 

1 3 

2 3 

1 1 
1 1 
5 5 

3 3 

5 1 
1 1 
1 1 
1 1 
2 2 
2 2 

6 10 
2 1 
7 10 



1 1 
1 1 
8 2 

4 3 
1 1 

3 4 
1 2 
1 1 

1 3 

7 3 

2 3 
1 1 

5 1 
1 1 

4 5 

6 6 

3 1 

2 1 
7 2 

1 5 

3 9 

1 1 

1 3 

2 1 
2 

- 4 

- 9 



1 1 
3 1 

5 6 
2 2 

1 7 

3 4 

3 6 

1 1 
1 2 
7 4 

3 2 

3 1 

1 1 

3 1 

2 1 

4 7 

6 5 
2 2 
2 - 
2 - 
2 - 
1 - 
4 

1 - 
3 - 

1 - 
6 - 
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lABLE 4 (Continued) 



Question 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 
29 



FI F2 

12 2 2 

3 2 11 

5 6 7 6 

2 3 3 1 

11 6 2 

3 1 2 1 

3 1 11 

12 11 

11 16 

7 5 6 4 

3 1 12 

3 1 11 

11 11 

3 1 12 

2 1 1 1 

4 4 3 4 

6 3 11 

2 1 11 

2 1 3 1 

2 2 1 3 

2 1 3 - 

11 1 - 

4 1 2 - 

11 1 - 

3 2 1 - 

14 4 - 

6 5 5 - 



II 12 

13 3 2 

2 6 3 1 

3 5 3 2 

11 2 4 

11 6 1 

4 6 4 7 

15 2 2 

11 12 

11 12 

6 8 5 3 

11 14 

13 11 

11 5 3 

4 1 15 

14 2 1 

3 1 2 1 

11 12 

2 5 3 1 

4 - 5 2 

3 - 11 

4 - - 4 

1 - - 1 

1 - - 

3 - - 

2 - - 

3 - - 

6 - - - 



Participant 
G1 G2 HI H2 
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TABLE 5 

NUMBER OF SUBROUTINES REQUIRED PER QUESTION BY PARTICIPANTS 



Question 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 
29 



Participant 
B1 B2 Cl C2 



A1 A2 

2 2 
1 1 
2 4 

1 2 

2 5 
1 1 

1 3 
1 1 

2 2 
2 2 
1 2 

3 3 

1 1 
2 1 
2 1 
3 2 
1 1 
1 2 
1 3 

1 



1 2 
2 1 

4 2 
1 1 

5 2 
1 1 

1 4 
1 1 

2 2 

3 3 

1 2 
1 2 

2 4 
2 1 
2 1 

3 2 

3 3 

2 2 
1 1 
2 2 
5 3 

2 - 



2 2 
1 2 
1 2 
1 1 
2 2 
2 1 
1 2 
1 1 
1 1 
3 1 

1 1 

2 3 

1 3 
1 1 
1 2 
1 1 
1 1 
1 1 

3 4 

2 2 
1 1 
1 1 
1 1 
2 1 
2 1 
1 1 
2 6 
1 1 

4 4 



D1 D2 

1 1 
1 1 
3 3 

1 1 
1 2 

1 3 
1 1 
1 1 

2 3 
6 2 
1 1 
1 2 
1 1 
1 1 
1 1 

3 4 
1 1 
1 1 

4 2 

2 2 
2 4 

1 1 
1 3 

1 2 

1 

2 

- 3 



El E2 



1 1 
2 1 
5 5 

1 2 
1 8 
1 1 
1 4 

1 1 

1 3 

7 3 

1 2 
5 2 

1 1 

2 1 

1 1 
4 2 

3 2 

1 2 
1 - 
1 - 
2 - 
1 - 

3 - 
1 - 
2 - 
1 - 

4 - 



TABLE 5 (Continued) 



Question 


FI 


F2 


Participants 
61 62 HI 


H2 


11 


12 


1 


1 


2 


2 


2 


1 


2 


1 


1 


2 


2 


2 


1 


1 


2 


2 


1 


1 


3 


5 


2 


4 


5 


3 


2 


2 


2 


4 


1 


2 


2 


1 


1 


1 


2 


1 


5 


1 


2 


7 


2 


1 


2 


4 


3 


6 


1 


1 


1 


1 


3 


2 


3 


5 


7 


1 


1 


1 


1 


1 


2 


1 


1 


8 


1 


1 


1 


1 


1 


1 


1 


2 


9 


1 


2 


1 


5 


1 


1 


2 


2 


10 


7 


4 


7 


4 


6 


5 


3 


3 


11 


1 


1 


1 


1 


1 


1 


1 


1 


12 


5 


2 


1 


2 


2 


3 


1 


1 


13 


1 


1 


1 


2 


1 


1 


3 


2 


14 


2 


1 


1 


2 


3 


1 


1 


3 


15 


1 


1 


1 


1 


1 


2 


1 


1 


16 


4 


2 


2 


5 


3 


1 


2 


2 


17 


3 


2 


1 


1 


1 


1 


1 


1 


18 


1 


1 


1 


1 


2 


3 


1 


1 


19 


1 


1 


1 


1 


2 


- 


4 


2 


20 


1 


1 


1 


1 


2 


- 


1 


2 


21 


2 


1 


3 


- 


2 


- 


- 


2 


22 


1 


1 


1 


- 


1 


- 


- 


1 


23 


3 


1 


1 


- 


1 


- 


- 


- 


24 


1 


1 


2 


- 


3 


- 


- 


- 


25 


2 


1 


1 


- 


1 


- 


- 


• 


26 


1 


2 


2 


- 


1 




- 


• 


27 

28 


4 


1 


2 


- 


2 


- 


- 


- 


29 


- 


- 


- 


- 


- 


- 


- 


- 






a 
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TABIaE 6 

NUMBER OF TIMES EACH PARTICIPANT USED EACH 
SUBROUTINE ON QUESTIONS 1-18 



Subroutines 



Partlclnants 


A 


B 


E 


F 


6 


H 


I 


Total 


A1 


2 


2 


- 


15 


2 


- 


8 


29 


A2 


4 


2 


2 


13 


4 


- 


11 


36 


B1 


4 


4 


1 


19 


3 


- 


6 


37 


B2 


7 


2 


3 


11 


5 


- 


8 


36 


Cl 


o 


2 


- 


10 


1 


- 


8 


24 


C2 


2 


2 


- 


11 


3 


- 


10 


28 


D1 


2 


2 


2 


15 


1 


- 


6 


28 


D2 


3 


3 


3 


13 


3 


- 


5 


30 


El 


3 


3 


1 


18 


5 


2 


7 


39 


E2 


6 


3 


2 


11 


5 


- 


15 


42 


FI 


3 


3 


1 


18 


5 


2 


7 


39 


F2 


5 


4 


- 


12 


4 


- 


5 


30 


61 


2 


4 


- 


13 


2 


- 


15 


36 


62 


5 


2 


1 


17 


2 


- 


11 


38 


HI 


3 


2 


- 


16 


4 


3 


5 


33 


H2 


1 


3 


- 


15 


2 


3 


9 


33 


11 


2 


3 


- 


13 


2 


1 


10 


31 


12 


2 


4 


- 


9 


4 


2 


12 


33 


Total Trained 


24 ^ 


25 


5 


137 


25 


8 


72 


296 


Percent of Total 


8.1 


8.4 


1.7 


46.3 


8.4 


2.7 


24.3 




Total Untrained 


35 


25 


11 


112 


32 


5 


86 


306 


Percent of Total 


11.4 


8.2 


3.6 


36.6 


10.4 


1.6 


28.1 





o 



APPENDIX B 

COVER LETTER FOR REFERENCE QUESTION RATING FORMS 
REFERENCE QUESTION RATING FORM A 
REFERENCE QUESTION RATING FORM B 
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LIBRARY RESEARCH CENTER 
428 Library 



UNIVERSITY OF ILLINOIS 
Urbana, Illinois 



COVER LETTER FOR REFERENCE QUESTION RATING FORMS 



July 20, 1966 



A few weeks ago you were kind enough to cooperate with a research 
project I am undertaking by supplying some information for me. 

Thank you very much; the information has been useful. Now, may I 
ask you to be of service to the study again? 

I am attempting to construct a set of "fact" type reference questions 
typical of those asked in medium-sized public libraries in the 
Midwest. Would you have one of your senior reference staff members 
take each set of questions enclosed and rate each question as to its 
relative difficulty? I realize that it is very hard to judge how 
difficult it would be to answer a reference question without actually 
attempting to answer it and that there is probably no such thing as 
a typical reference question. However, I think I will be able to 
have more confidence in my set of questions if I can get the judgment 
of practicing reference librarians on the potential questions. 

May I suggest that the librarians consider these questions in com- 
parison with the numerous questions for specific facts that they 
answer day after day (leaving out of consideration the often more 
difficult and complicated questions calling for interpretation or 
judgment)? Then, on the basis of such factors at the probable time 
required to produce an answer, detailed knowledge of sources required, 
and probability of success in answering, they can rate each question 
as average or typical among the factual questions asked your reference 
department, easier or simpler than average, or more difficult or 
complex than typical. Verbal comments concerning individual questions 
or the set as a whole are welcome and can be written on the back of 
the rating form or on a separate sheet. 

I would appreciate very much your returning the completed rating forms 
as soon as convenient in the stamped self-addressed envelope. Return 
by August 15 would be particularly helpful. 



Sincerely 



CAB/sp 

enclosure 



Charles A. Bunge 
Research Associate 
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UNIVERSITY OF ILLINOIS 
Urbana, Illinois 



LIBRARY RESEARCH CENTER 
428 Library 



REFERENCE QUESTION RATING FORM A 

Easier Average Harder 



1, At what age does a boy cease to be 
called "master”? 

2. Medical symbol for male and female. 

3. Is American Indian Day a National 
Holiday? 

4. Number of weeks it takes for different 
common vegetables to produce ripe 
produce . 

5, How is the date of Easter determined? 
On what dates will Easter fall from 
1967 to 1970? 

6, I want a list of wholesalers of 
tropical fish in Miami} Florida. 



7. How long do you have to wait in 
Kansas after a divorce before a second 
marriage? 

8. What is the weight of sand per cu. ft, 
or cu. yd.? 

9. What are the top ten high schools in 
the U.S,, according to the number 

of National Merit Scholarship 
finalists? 

10. Was there a real Indian who modeled 
for the U.S, nickel? If so, what is 
or was his name? 

11. Is the life expectancy in the U«S, 
different for whiles and non-whites? 

How is this computed? 

12. On which concerto was "Tonight We 
Love" based? 

13. How do I get the address of an American 
Soldier who is serving in Viet Nam? 
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Easier Average Harder 



14. How does one play Red Dog? 

15. What is the pension for ex-presidents 

of the U.S. and widows thereof? 

16. What kind of fishing license does an 
out-of-state tourist need for a week 

in Missouri? How much does it cost? 

17. Is it legal to photograph money for 

advertising? — 

18. What is the height of the basket in 

basketball for high school? __ 

19. What is the average salary of a stock 

broker? _ 

20. Where can I locate a map of the travels 

of Ulysses? _ 

21. What was the nature of the change made 

in the Cost of Living Index in 
January, 1964? _ 

22. What was the source of the phrase 

"Voice of the Turtle"? _ 

23. Where can I find the names and addresses 

of all U.S. and state senators and 
congressmen currently representing this 
locality? _ 

24. What is the location of Lake Taal? __ 

25. Whom does one contract to buy land 
from the U.S. Government? 

26. I need a copy of the building code 
for this city and state. 

27. How high are the tides around 
Inchon, Korea? 

28. Where can I find a list of 20 

homonyms? 

29. Where is a list of nations newly 

independent since 1957? 
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Easier Average Harder 



30. What is the comparison of the value 
of the dollar for 1916 and 1960? 

31. Definition of "Taliesin" 

(F, L. Wright's home)? 

32. What or who is the source of the 
"debate" about how many angels can 
stand on the point of a needle? 

33. What was the name of Hopalong 
Cassidy's horse? 

34. What now stands where Solomon's 
Temple once stood? 

35. Whom did Henry VIII 's last wife 
marry after Henry's death? Was 
there a child of the marriage? A 
boy or a girl? 

36. What are the percentages of Roman 
Catholics and Protestants in West 
Germany? 

37. What is the gestation period of 
the rat? 

38. What is the time differential 
between Viet Nam and New York? 

39. What are the different hands for 
on a barometer? How do I set 
them for this locality? 

40. A list of publishers of Catholic 
religious educational print and 
non-print materials. 



o 
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LIBRARY RESEARCH CENTER UNIVERSITY OF ILLINOIS 

428 Library Urbana, Illinois 



REFERENCE QUESTION RATING FORM B 

Easier Average Harder 



1. Where can I find a list of publishers 
of true romance magazines? 

2. What are the colors of the church altar 
cloths for the different seasons? 

3. What is the cost per capita of the 
U.S. space program? 

4. What are the five U.S. states with 
the largest population, in order 
of size? 

5. Where can I find the letter of Martin 
Luther King written in the Birmingham 
Jail? 

6. Where are current editions of the 
U.S. and state constitutions? 

7. How much more expensive is it for a 
family to live in Chicago than in 
Atlanta? 

8. What is the present membership in 
U.S. churches and what are recent 
trends in church attendance? 

9. What are the state and local laws 
regarding passing on the left when 
driving a motor vehicle? 

10. What were the prices of bread and 
butter in the 1920 's? 

11. Who is the present head of the Congo 
government? 

12. Which cities in the U.S. have the 
largest police forces? 

13. Can a four -family apartment house 
legally be built in the (X) area of 
this town? (X being one or another 
section) 
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Easier Average Harder 



14. I am a male. Where can X complete a 
course in practical nursing? 

15. What Is one acre equal to In the 
metric system? 

16. What Is meant by the "battered 
child syndrome"? 

17. Are the holidays of July 4th 
and May 30th national holidays, 
that Is, do all states observe 
them? 

18. Where can I find the name and 
address of the company that made a 
higli chair with the brand "Comfort 
Lines"? 

19. How do you word wedding Invitations 
when the bride's parents are 
divorced and the mother has re- 
married? 

20. What Is the annual rainfall for 
the principal cities of the world? 

21. What part of the castor bean plant 
Is poisonous and what would be the 
first aid for such poisoning? 

22. What Is the life span of the Red 
Squirrel? 

23. Where Is a list of the leading 
U.S. gold and silver mines? 

24. What Is the recommended gap for 
points on a 1962 Falcon? 

25. What are cultured pearls? 

26. How much raw materials will be 
needed for a concrete project of 
3" X 100 sq. ft.? How much would 
this much concrete weigh? 

27. What was the size of the old 
one dollar bills of before 
1929? 
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Easier Average Harder 



28. Who is credited with founding the 
"Impressionistic" school of music? 

29. What are the dimensions of a baseball 
diamond for Little League play? 

30. What are the locations of camping 
trailer parks in Wyoming? 

31. Where can I locate some examples 
of Haikai or Haiku? 

32. What is the milage from Chicago 
to Hongkong? 

33. Where is a biography of Admiral Kidd 
of World War II? 

34. When did Hof fa affear before the 
Congressional Committee hearings 
conducted by Senator McClellan? 

35. How many soldiers were in a regiment 
during the Civil War? 

36. Was Wildflecken in the American 
zone of occupation? 

37. How high above sea level is 
Cleveland? 

38. What is the difference between term 
insurance, ordinary life, and 
endowment insurance? 

39. What would be the amount of interest 
paid on $50,000 for 20 years at 
51/2 percent? 

40. Has the average height and weight of 
Americans increased since the turn 
of the century? 
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APPENDIX C 



TEST QUESTIONS AS ASKED, WITH 



* 



"BEST" ANSWERS 



1. I would like the names and office addresses of the Senators and 
Representatives representing me in the State Legislature. I live in the 
downtown area of this city. Now I want the same thing for the Federal 
Leg? ilature. 

(Varied from library to library) 

2. I head somewhere that California passed New York in population, to 
become the most populous state ^ since the 1960 census. Could you find 
me the population of California and New York later than 1960 to verify 
this? 

(1965 estimate; New York, 18,075,000; California, 18,605,000) 

3. For a nature project that I'm involved with, I need to know the 
average life span of the Red Squirrel. 

(Average or normal, 6 or 7 years; potential, 10 years) 

4. For a speech that I am preparing I need to know the total cost of 
the U.S, space program for a recent year. Then I want to convert it 
j.nto a per capita figure. 

(1965; Approximate expense, $6,886,000,000; estimated population, 
193,818,000 - $35.52 per capita) 

5. Recently a subject mentioned as a possible discussion topic at my 
club was the "battered child syndrome." This term is new to me. Could 
you find just enough information on this so that I could know to what 
it refers? 

(The group of symptoms which indicates to a doctor that a child has 
been beaten or mistreated) 

6. For a Sunday School lesson I am preparing I need to know what now 
stands on the spot where Solomon's Temple once stood. 

(Islamic sacred edifice, the Dome of the Rock) 

7. How many soldiers would have been in a typical or average regiment 
in the Civil War? 

(Union side: Authorized at 845-1050 men; were actually usually much 

smaller) 

8. On what dates will Easter Sunday fall in 1967 and 1968? 

(1967: March 26; 1968: April 14) 

9. Can you find me some examples of Haikai or Haiku? 

(Form of Japanese poetry; can be found in various places) 

10. What was the name of Hopalong Cassidy's horse? 

(Topper) 



* Sources for answers are not given because most questions were answered 
from various sources. 
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APPENDIX C (continued) 

11. I have a baby bed which needs to have the teething rail replaced. 

All I can find on it is the name "Childcraf t." Can you find me an address 

for the company who made this? 

(Smith Cabinet Manufacturing Company, Salem, Indiana) 

12. Can you find me a copy of the letter Martin Luther King wrote in the 
Bimingham Jail? I think it has become sort of a Civil Rights classic. 

(Can be found in magazines, civil rights anthologies, biographies of 

King) 

13. When, that is, at what age should a boy cease to be called "master"? 

(10 or 12 years old) 

14. A neighbor has Castor Beans in his garden. Is a part of the Castor 

Bean plant poisonous? If so, what part? 

(All parts of the plant are poisonous, seeds are deadly to children) 

15. If one had a child who did get poisoned from a Castor Bean, what 
should be done— what might be the best first aid? 

(Call the doctor; call the poison control center ; induce vomiting) 

16. I have been given a barometer. It is one of the round-faced kind 
with two needles. What are these needles for? How do I make the initial 
setting for this area? 

(Black needle is indicator; gold needle is reference hand; call radio 
station or weather bureau for initial setting) 

17. I am wanting to build a patio. It is to consist of a concrete slab 
of 6 inches by 100 square feet. How much raw materials will I need for 

such a project? 

(12.5 sacks cement, 25 cubic feet sand, 37.5 cubic feet gravel, 69 
gallon water) 

18. How much more or less expensive is it for an average family to live in 
Chicago than it is in Atlanta? 

(1963: Chicago, 102.0; Atlanta, 93.9; based on Washington, D.C.=100) 

19. I am shopping for a used air conditioner. Some have their size ratings 
giving in B.T.U.'s and some in Tons. What is the relationship between these 

two ratings? 

(1 Ton = approxim.ately 12,000 B.T.U.) 

20. When did Jimmy Hoffa first appear and testify before the Congressional 
committee hearing conducted by Senator McClellan? 

(August, 1957) 

21. Somewhere in my reading I have run across the name of an Admiral Kidd. 
I'd like just enough biographical information to know why he would have 
been written about. As I remember he made his name in the World War II era 

of the late 1930"s or early 1940"s. 

(Rear Admiral of U.S, Fleet, killed during Japanese attack on Pearl 

Harbor, 1941) 

22. What is the recommended gap for points on a 1962 Ford Falcoln six- 

cylinder sedan? 

(.025) 
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APPENDIX C (continued) 



23. Around 2-1/2 or three years ago there was a change in the way of cost 
of living index is computed. What, basically, did this change consist of? 

(Up-dated sample of cities, stores, etc.; modernized list of goods and 
services; revised weights to reflect modern spending patterns) 

24. What percentage of the population of West Germany is Roman Catholic 
and what percentage is Protestant? 

(1961; 51% Protestant; 44.1% Roman Catholic) 

25. 1 am going on vacation in Missouri for a week. What kind of fishing 
license do I need for a short time and how much does it cost? 

(14 day tourist license costs $3.25; also need $2.25 trout permit) 

26. I am thinking about raising some vegetables. How long does it take 
for various common vegetables to produce ripe produce? 

(Answer varied according to vegetables chosen) 

27. Was Wildflecken in the American zone of occupation? 

(Yes) 

28. Whom should I contact if I wish to investigate buying land from the 
U. S» Government? 

(Bureau of Land Management, Washington, D. C., or Regional Land Offices) 

29. Can you find me a map showing the travels of Ulysses? 

(Can be found in atlas of classical world) 



APPENDIX D 

OBSERVATION SCHEDULE /INTERVIEW GUIDE 
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Llbrary Research Center, University of Illinois 

Reference Study 
Observation Schedule 

Question No.: Librarian: 



Step No. 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10+ 


Activity 

USED REFERENCE TOOL: 
As Ans. Soui. e 






















Found Answer? 






















As Guide to Source 
Periodical Index 






















Other Index 






















Bib. or Gd. to Lit. 






















Other 






















Found Lead? 






















As Source of Def. 






















USED CARD CATALOG: 
Under Subiect 






















Under Type of Work 






















Under Specific Work 






















Found Lead? 






















USED NON-REFERENCE WORK: 
Went Directly to 






















Via Card Catalog 






















Found Answer? 






















USED PERIODICAL: 
Went Directly to 






















Via Index 






















Found Answer? 






















CONSULTED PATRON: 

* Make Oues. More Specif. 






















Specify General Subiect 






















Define Term(s) 






















Specify Kind of Info. 






















Suitability of Answer 























Correct Answer Found? Yes: No: Time: 



No. of Sources Consulted: Ans. Source: 

(If not ans.) Is ans. source in collection? Yes: 



Notes on Steps: 



-99- 



LIBRA..Y RESEARCH CENTER, UNIVERSITY OF ILLINOIS 

Reference Study 

Guide to Interview Concerning Answering Process 
Question No.: Librarian: — 

1. Can you describe briefly what sort of thoughts and decisions started 
through mind as you decided how to attack this question? 



2. Were there any particular parts or words of the question that you 
considered especially important, as you thought about how you would 
locate the answer? If so, what were they? 



3. When you first turned to begin looking for the answer would you say 
that you were mainly thinking of a certain kind of book or tool, for 
example, an alamanac; a general or specific subject, such as science 
or astronomy; or a specific tool, by title, location, etc.; or do any 
of these choices cover your intentions? (If a choice is made, ascertain 
which type, subject, work, etc.) 



shelves 

4. When we started for the catalog did you have a specific source 

other (specify)__ - — . 

in mind to try, or did you hope to find a likely source listed or 
shelved in the appropriate subject section? (Ask this for each 
appropriate step. If answered "specific source," ascertain which. ) 



5. Can you tell my why the sources you first thought about might have come 
to mind? (If needed for explanation: For example, have you answered 

similar questions from these works in the past?) 



6. Can you tell me what might have caused you to choose to try the 
specific source you first used as a possible answering source? 



Do you remember studying any or all of the sources you consulted to 
answer this question in any of the institutes, courses or other 
training you have had. 



7 . 
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AFPENDIX E 



INSTRUCTIONS TO PARTICIPATING STAFF MEMBERS 



As you have probably been told, I am with the Library Research 



1 



Center of the Graduate School of Library Science at the University 



of Illinois. I am conducting a study into just how practicing 
public library reference staff members go about finding the answers 
to questions asked by patrons. As one method of doing this, I am 
going to ask you a set of reference questions and watch closely as 
you go about finding the answers to them. Some of the questions are 
fairly simple, and others are more diff' not even be 



answerable at this library, since I have not checked the questions 
against the collection. I would like you to treat me and my questions 
exactly as you would a patron from off the street. Ask me questions 
when and if you would a patron, tell me you can’t find the information 
if t at is what you would tell a patron, suggest trying another library 
or borrowing a book on inter library loan, and so forth. Work as 
though you are on duty alone and cannot ask another librarian for 
help. Also, work as though I am a patron whom you would not expect 
to use the card catalog and books alone. I shall be following you 
around making notes. Please do not allow this to make you uncomfortable. 
This is not done in an effort to judge you, but to keep me from 
forgetting what I have observed, since I want to be able to report 
what actually happens in public library reference work. Please work 
as rapidly as you would if your patron were waiting on the telephone 
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APPENDIX E (continued) 



or were evidently in somewhat of a hurry, but not so quickly as to do 
less thorough work than you usually do. There is no specific number 
of questions that we must finish, but I do want to know how long it 
takes a person with your experience to answer such questions. Are 
there any questions? 
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